I have a data set with some brand names. By preprocessing (e.g lowercasing, removing stopwords, triming the whitespace, string splitting, etc.) I got from 320 distinct cases in the beginning to 114. However, there is still some room for improvement.
# A tibble: 6 x 3
record Colum Name
<dbl> <chr> <chr>
1 2 A check24
2 3 A NA
3 7 A weriwox
4 9 A verifox
5 10 A chec24
6 11 A smava
As you can see, case 3 is similar to case 4 and case 1 is similar to case 5. For case 6 there is no equivalent. Is there a way to successively standardize the strings by replacing the similar ones?
An idea is to use the tidystringdist/stringdist package. Here I can calculate a distance measure for the string (e.g. Levenstein distance). From here I could unify those with low distance measures (e.g. osa < 3).
library(dplyr)
library(tidystringdist)
comb = tidy_comb_all(d, Name)
tidy_stringdist(comb, method = c("osa")) %>%
filter(osa <= 2)
# A tibble: 6 x 3
V1 V2 osa
<chr> <chr> <dbl>
1 check24 chec24 1
2 check24 check42 1
3 check24 chevk24 1
4 check24 check34 1
5 check24 schek24 2
6 check24 scheck24 1
For example, based on the results I could get rid of "chec24" and "check42", by replacing it with "check24". So I could reduce the number of names even further. The goal is not the get the "true" name match (because I have no list with the real names). It´s more about replacing the very similar ones to reduce the overall number of distinct names. However, I have no idea who to do this... maybe a while loop.
Here is a small dput of the similar names:
structure(list(record = structure(c(2, 3, 7, 9, 10, 11, 16, 22,
39, 59, 89, 110, 113, 148, 167, 183, 184, 203, 270, 352, 399,
1243, 1254, 1258, 1274, 1299, 1309, 1321, 1331, 1344, 1352, 1375,
1383, 1416, 1417, 1421, 1484, 1493, 1555, 1571, 1575, 1588, 1590,
1591, 1644, 1649, 1672, 1676, 1744, 1770, 1816, 1822, 1906, 1934,
2080, 2090, 2335, 2374, 39, 148, 1299, 1334, 1487, 1681, 1816,
2159, 2335, 2376, 2376, 148, 1254, 1334, 1254, 1254, 22, 25,
29, 34, 59, 69, 75, 121, 122, 138, 148, 270, 706, 1230, 1274,
1276, 1344, 1357, 1367, 1375, 1376, 1383, 1386, 1421, 1422, 1468,
1486, 1493, 1507, 1715, 1743, 1790, 1822, 1837, 2159, 52, 1337,
1386, 2257, 1241), label = "record: Record number", format.spss = "F7.0", display_width = 7L),
Colum = c("A", "A", "A", "A", "A", "A", "A", "A", "A", "A",
"A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A",
"A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A",
"A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A",
"A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A",
"B", "B", "B", "B", "B", "B", "B", "B", "B", "B", "C", "D",
"D", "D", "E", "F", "K", "K", "K", "K", "K", "K", "K", "K",
"K", "K", "K", "K", "K", "K", "K", "K", "K", "K", "K", "K",
"K", "K", "K", "K", "K", "K", "K", "K", "K", "K", "K", "K",
"K", "K", "K", "L", "L", "L", "L", "M"), Name = c("check24",
NA, "weriwox", "verifox", "chec24", "smava", "verivox", "targobank",
"facebook", "kredit24", "taunus", "credit24", "finazcheck",
"maxda", "diba", "norisbank", "scout24", "easycredit", "kredit",
"finanzcheck", "swawa", "check42", "barclaycard", "kredit24.de",
"idealo", "immo", "santander", "wüstenrot", "check24.de",
"citybank", "smawa", "smama", "preisvergleich", "smarva",
"finanzscheck", "parisba", "versch", "comdirect", "ing",
"fakten", "tau res", "chevk24", "24 check", "veritas", "verivox check24 dr. klein",
"verivix.de", "easy", "easy-credit", "finanzsvout24", "check34",
"consorfinanaz", "sparkasse", "schek24", "ingdiba", "dr. klein",
"smavaa", "commerz", "sparda-bank", "youtupe", "creditplus",
"scout", "smavey", "interhyp", "krediten", "oostbank", "comdirekt",
"bank", "verivox.de", "preisvergleich24", "auxmoney", "hanseatikbank",
"klein", "advanzia", "americanexpress", "postbank", "targo",
"google", "kredit.de", "maxx", "moneycheck", "financescout24",
"yerivox", "finanzscout", "kreditvergleich.de", "volkswagen",
"vergleichskredit", "scheck24", "finanzscout24", "creditvergleich",
"preisvergleiche.de", "barclay", "toptarif", "finanztest",
"chech24", "dr. weber", "online-aspecte.de", "finanzcheck.de",
"targa", "checl24", "verybox", "verifox.de", "finanzen",
"n24", "finanzcheck24", "testsieger", "online", "volksbank",
"checkkredit.eu", "n26", "kreditvergleich", "finanzvergleich",
"kreditexperte.de", "kreditmacher", "samba")), class = c("tbl_df",
"tbl", "data.frame"), row.names = c(NA, -114L))
