Discovery of complex glitch patterns : a novel approach to quantitative data cleaning Berti-Equille Laure auteur aut IRD Dasu T. auteur aut IRD Svrivastava D. auteur aut IRD text bookSection eng text/pdf born digital access Quantitative Data Cleaning (QDC) is the use of statistical and other analytical techniques to detect, quantify, and correct data quality problems (or glitches). Current QDC approaches focus on addressing each category of data glitch individually. However, in real-world data, different types of data glitches co-occur in complex patterns. These patterns and interactions between glitches offer valuable clues for developing effective domain-specific quantitative cleaning strategies. In this paper, we address the shortcomings of the extant QDC methods by proposing a novel framework, the DEC (Detect-Explore-Clean) framework. It is a comprehensive approach for the definition, detection and cleaning of complex, multi-type data glitches. We exploit the distributions and interactions of different types of glitches to develop data-driven cleaning strategies that may offer significant advantages over blind strategies. The DEC framework is a statistically rigorous methodology for evaluating and scoring glitches and selecting the quantitative cleaning strategies that result in cleaned data sets that are statistically proximal to user specifications. We demonstrate the efficacy and scalability of the DEC framework on very large real-world and synthetic data sets specialized RESEAU INFORMATIQUE TRAITEMENT DE DONNEES ERREUR METHODE D'ANALYSE ANALYSE STATISTIQUE 122APPLIC Proceedings of the 27th international conference on data engineering Abiteboul S. ed. edt IRD Böhm K. ed. edt IRD Koch C. ed. edt IRD Kian Lee Tan ed. edt IRD 733-744 2011 ICDE.International Conference on Data Engineering, 27., Hanovre (DEU), 2011/04/11-16 IEEE Conference Publication https://www.documentation.ird.fr/hor/fdi:010055317 10.1109/ICDE.2011.5767864 978-1-4244-9194-0 [F B010055317] https://www.documentation.ird.fr/hor/fdi:010055317 https://www.documentation.ird.fr/intranet/publi/depot/2012-05-23/010055317.pdf Accès réservé (Intranet de l'IRD) IRD - Base Horizon / Pleins textes 2012-05-22 2023-02-22 fdi:010055317 fre