High Dimensional, Robust, Unsupervised Record Linkage

• Autorzy: Sabyasachi Bera , Snigdhansu Chatterjee
• Języki publikacji: EN

Abstrakty

EN

We develop a technique for record linkage on high dimensional data, where the two datasets may not have any common variable, and there may be no training set available. Our methodology is based on sparse, high dimensional principal components. Since large and high dimensional datasets are often prone to outliers and aberrant observations, we propose a technique for estimating robust, high dimensional principal components. We present theoretical results validating the robust, high dimensional principal component estimation steps, and justifying their use for record linkage. Some numeric results and remarks are also presented. (original abstract)

Słowa kluczowe

EN

Principal Component Analysis (PCA); Bayesian models

PL

Analizy głównych komponentów; Modele bayesowskie

• Czasopismo: Statistics in Transition
• Rocznik: 2020
• Tom: 21
• Numer: nr 4 Special Issue
• Strony: 123--143

Twórcy

autor

Sabyasachi Bera

• University of Minnesota, USA

autor

Snigdhansu Chatterjee

• University of Minnesota, USA

Bibliografia

• CHRISTEN, P., (2011). A survey of indexing techniques for scalable record linkage and deduplication. IEEE transactions on knowledge and data engineering, 24(9), pp. 1537-1555.
• DRACHSLER, H., BOGERS, T., VUORIKARI, R., VERBERT, K., DUVAL, E., MANOUSELIS, N., BEHAM, G., LINDSTAEDT, S., STERN, H., FRIEDRICH, M., et al., (2010). Issues and considerations regarding sharable data sets for recommender systems in technology enhanced learning. Procedia Computer Science, 1(2), pp. 2849- 2858.
• ENAMORADO, T., FIFIELD, B., and IMAI, K., (2018). Using a probabilistic model to assist merging of large-scale administrative records. Available at SSRN 3214172.
• FANG, K.-T., KOTZ, S., and NG, K.-W., (1990). Symmetric Multivariate and Related Distributions. CRC Press.
• FATEMI, B., KAZEMI, S. M., and POOLE, D., (2018). Record linkage to match customer names: A probabilistic approach. arXiv preprint arXiv:1806.10928.
• FELLEGI, I. P. and SUNTER, A. B., (1969). A theory for record linkage. Journal of the American Statistical Association, 64(328), pp. 1183-1210.
• HAN, Y. and LAHIRI, P., (2019). Statistical analysis with linked data. International Statistical Review, 87, pp. S139-S157.
• HERZOG, T. N., SCHEUREN, F. J., and WINKLER, W. E., (2007). Data quality and record linkage techniques. Springer Science & Business Media.
• JARO, M. A., (1989). Advances in record-linkage methodology as applied to matching the 1985 census of tampa, florida. Journal of the American Statistical Association, 84(406), pp. 414-420.
• LAHIRI, P. and LARSEN, M. D., (2005). Regression analysis with linked data. Journal of the American Statistical Association, 100(469), pp. 222-230.
• LARSEN, M. D. and RUBIN, D. B., (2001). Iterative automated record linkage using mixture models. Journal of the American Statistical Association, 96(453), pp. 32-41.
• LI, J., DOU, Z., ZHU, Y., ZUO, X., and WEN, J.-R., (2020). Deep cross-platform product matching in e-commerce. Information Retrieval Journal, 23(2), pp. 136-158.
• LISEO, B. and TANCREDI, A., (2013). Some advances on Bayesian record linkage and inference for linked data. URL http://www.ine.es/e/essnetdi_ws2011/ppts/Liseo_Tancredi.pdf.
• MARTINS, B., (2011). A supervised machine learning approach for duplicate detection over gazetteer records. In International Conference on GeoSpatial Sematics, pp. 34-51, Springer.
• NEWCOMBE, H. B. and KENNEDY, J. M., (1962). Record linkage: making maximum use of the discriminating power of identifying information. Communications of the ACM, 5(11), pp. 563-566.
• PAUL, D., (2007). Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statistica Sinica, 17(4), pp. 1617-1642.
• RONG, S., NIU, X., XIANG, E. W., WANG, H., YANG, Q., and YU, Y., (2012). A machine learning approach for instance matching based on similarity metrics. In International Semantic Web Conference, pp. 460-475, Springer.
• SADINLE, M., (2017). Bayesian estimation of bipartite matchings for record linkage. Journal of the American Statistical Association, 112(518), pp. 600-612.
• SADINLE, M. and FIENBERG, S. E., (2013). A generalized fellegi-sunter framework for multiple record linkage with application to homicide record systems. Journal of the American Statistical Association, 108(502), pp. 385-397.
• SALAS, J., (2019). Sanitizing and measuring privacy of large sparse datasets for recommender systems. Journal of Ambient Intelligence and Humanized Computing, pp. 1-12.
• SCHEUREN, F. and WINKLER, W. E., (1993). Regression analysis of data files that are computer matched. Survey Methodology, 19, pp. 39-58.
• SCHEUREN, F. and WINKLER, W. E., (1997). Regression analysis of data files that are computer matched-ii. Survey Methodology, 23, pp. 157-165.
• SHABTAI, A., ROKACH, L., and ELOVICI, Y., (2013). Occt: A one-class clustering tree for implementing one-to-many data linkage. IEEE Transactions on Knowledge and Data Engineering, 26(3), pp. 682-697.
• SINGLA, P. and DOMINGOS, P., (2006). Entity resolution with markov logic. In Sixth International Conference on Data Mining (ICDM'06), pp. 572-582, IEEE.
• SLOKOM, M., (2018). Comparing recommender systems using synthetic data. In Proceedings of the 12th ACM Conference on Recommender Systems, pp. 548-552.
• STEORTS, R. C. et al., (2015). Entity resolution with empirically motivated priors. Bayesian Analysis, 10(4), pp. 849-875.
• STEORTS, R. C., HALL, R., and FIENBERG, S. E., (2016). A bayesian approach to graphical record linkage and deduplication. Journal of the American Statistical Association, 111(516), pp. 1660-1672.
• STEORTS, R. C., VENTURA, S. L., SADINLE, M., and FIENBERG, S. E., (2014). A comparison of blocking methods for record linkage. In International Conference on Privacy in Statistical Databases, pp. 253-268, Springer.
• TASKINEN, S., KOCH, I., and OJA, H., (2012). Robustifying principal component analysis with spatial sign vectors. Statistics & Probability Letters, 82(4), pp. 765- 774.
• TORVIK, V. I. and SMALHEISER, N. R., (2009). Author name disambiguation in medline. ACM Transactions on Knowledge Discovery from Data (TKDD), 3(3), pp. 1-29.
• TREERATPITUK, P. and GILES, C. L., (2009). Disambiguating authors in academic publications using random forests. In Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries, pp. 39-48.
• VENTURA, S. L., NUGENT, R., and FUCHS, E. R., (2015). Seeing the non-stars:(some) sources of bias in past disambiguation approaches and a new public tool leveraging labeled records. Research Policy, 44(9), pp. 1672-1701.
• WANG, T., BERTHET, Q., and SAMWORTH, R. J., (2016). Statistical and computational trade-offs in estimation of sparse principal components. The Annals of Statistics, 44(5), pp. 1896-1930.
• ZHU, J., ZHANG, S., SINGH, L., YANG, G. H., and SHERR, M., (2016). Generating risk reduction recommendations to decrease vulnerability of public online profiles. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pp. 411-416, IEEE.

Identyfikatory

DOI

10.21307/stattrans-2020-034