Efficient Data Partitioning for Entity Resolution Systems

Abstract

Entity Resolution is the task of identifying duplicated records that refer to the same real-world entity. It is costly process that can take up to days for large datasets. Various Blocking Methods have been applied in Entity Resolution Systems to reduce the number of record pairs for comparison. It is still a big issue because a good blocking key is critical to the success of a blocking method and will ideally result in lots of small blocks. The efficiency of a blocking method is hindered by these large blocks since the resulting number of record pairs is dominated by the sizes of these large blocks. So, the researchers are still doing researches on handling the problems of large blocks. To overcome these problems, we would like to propose an efficient data partitioning system by introducing “Dynamic Block Based Structure” to enhance the blocking efficiency.