BTEDD: Block-level tokens for efficient data deduplication in public cloud infrastructures

Abstract

In today's digital era, the exponential growth of data necessitates effective storage and management solutions. The cloud has vast storage possibilities to store huge amounts of data. Public access to the cloud leads to duplicate copies of data stored in the storage. Maintaining a single copy of data in the cloud is most important for efficient data storage management. This paper introduces a groundbreaking strategy for improving the efficacy of cloud storage through innovative data deduplication techniques at the block levels. The block-level duplication verification efficiently identifies the duplicate data in the storage. It helps to protect the duplicate storage in the cloud data storage infrastructure. The block-level deduplication technique uses variable-length blocks based on the duplicate content of the block. Initially, A file is divided into a number of blocks with a size of 5kb. According to the proposed method, If any block is partially matched with a block already stored in the cloud, then that block is further divided into smaller blocks based on the matching percentage. The smaller blocks help to deduplicate the data more effectively. The work is implemented in a live cloud setting with a C# application hosted on MyASP.NET. The proposed methodology's effectiveness is validated against existing deduplication techniques. The results reveal a marked improvement in storage utilization and data management, affirming the potential of the approach to revolutionize cloud storage efficiency.