Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics
المؤلفون المشاركون
Binxing, Fang
Zhang, Xi
Yao, Yuntao
Ji, Yingsheng
المصدر
Mathematical Problems in Engineering
العدد
المجلد 2016، العدد 2016 (31 ديسمبر/كانون الأول 2016)، ص ص. 1-12، 12ص.
الناشر
Hindawi Publishing Corporation
تاريخ النشر
2016-10-18
دولة النشر
مصر
عدد الصفحات
12
التخصصات الرئيسية
الملخص EN
Detecting near duplicates on the web is challenging due to its volume and variety.
Most of the previous studies require the setting of input parameters, making it difficult for them to achieve robustness across various scenarios without careful tuning.
Recently, a universal and parameter-free similarity metric, the normalized compression distance or NCD, has been employed effectively in diverse applications.
Nevertheless, there are problems preventing NCD from being applied to medium-to-large datasets as it lacks efficiency and tends to get skewed by large object size.
To make this parameter-free method feasible on a large corpus of web documents, we propose a new method called SigNCD which measures NCD based on lightweight signatures instead of full documents, leading to improved efficiency and stability.
We derive various lower bounds of NCD and propose pruning policies to further reduce computational complexity.
We evaluate SigNCD on both English and Chinese datasets and show an increase in F1 score compared with the original NCD method and a significant reduction in runtime.
Comparisons with other competitive methods also demonstrate the superiority of our method.
Moreover, no parameter tuning is required in SigNCD, except a similarity threshold.
نمط استشهاد جمعية علماء النفس الأمريكية (APA)
Zhang, Xi& Yao, Yuntao& Ji, Yingsheng& Binxing, Fang. 2016. Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics. Mathematical Problems in Engineering،Vol. 2016, no. 2016, pp.1-12.
https://search.emarefa.net/detail/BIM-1112113
نمط استشهاد الجمعية الأمريكية للغات الحديثة (MLA)
Zhang, Xi…[et al.]. Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics. Mathematical Problems in Engineering No. 2016 (2016), pp.1-12.
https://search.emarefa.net/detail/BIM-1112113
نمط استشهاد الجمعية الطبية الأمريكية (AMA)
Zhang, Xi& Yao, Yuntao& Ji, Yingsheng& Binxing, Fang. Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics. Mathematical Problems in Engineering. 2016. Vol. 2016, no. 2016, pp.1-12.
https://search.emarefa.net/detail/BIM-1112113
نوع البيانات
مقالات
لغة النص
الإنجليزية
الملاحظات
Includes bibliographical references
رقم السجل
BIM-1112113
قاعدة معامل التأثير والاستشهادات المرجعية العربي "ارسيف Arcif"
أضخم قاعدة بيانات عربية للاستشهادات المرجعية للمجلات العلمية المحكمة الصادرة في العالم العربي
تقوم هذه الخدمة بالتحقق من التشابه أو الانتحال في الأبحاث والمقالات العلمية والأطروحات الجامعية والكتب والأبحاث باللغة العربية، وتحديد درجة التشابه أو أصالة الأعمال البحثية وحماية ملكيتها الفكرية. تعرف اكثر