Token-Based Data Cleaning Technique for Job Assigning System

Abstract

The problem of detecting and eliminating duplicate data is one of the major problems in broad area of data cleaning and data quality. Data cleaning is the process of identifying and removing duplicates in the database. Today, several existing data cleaning techniques have been widely used for this purpose. This paper describes how to effectively use token-based cleaning technique and positional algorithm in the development of job assigning system for detecting and eliminating duplicate job application forms. Token-based data cleaning technique is effective and efficient for data cleaning which decomposing and reassembling the data to get a unique set of string from ordinary user input data in comparing and eliminating duplicate records from database. This approach also eliminates the need to use the entire long string records with multiple passes, for duplicate identification. After eliminating duplicate data, system checks that the person has already assigned in one job position or not using Positional algorithm by identifying NRC number as match field. Positional algorithm is used because it can find errors with positional disorders and gap penalties. As a result, the system can get clean data that are not duplicated.