Using Hashing Algos to prevent file duplicates and save storage

This article aims to show a method of preventing duplicate files and saving storage space by using hashing algorithms.

There are cases where a video, image, or media might be trending due to a meme, news, or whatever can make anything trend. In such a scenario, the same file gets uploaded repeatedly by different users. With repeated uploads, the load on the server will increase, and more space used to save duplicates of the same file.

I’m going to use a method that makes use of a file store, a key-value store (hash-storage linker), and a reference object to a file.

It is responsible for saving the actual file uploaded by users. Storage platforms or services like S3, Google Storage, Block Storage, for example, can be used as a file store.

It is a DB that holds the file hash value as a key and the path to the file on the file store (e.g S3) as its value. A KV-store DB is great for storing this type of data.

It serves the middle-man between the file store and the file reference.

Provided the files exist, a hash-storage linker record must exist.

Related posts:

Someone is always asking about my “why,” as if I wake up at first light and say “This is why I will do this thing today.” I am occasionally surrounded by Rotarians (I married one) many of whom…

Thus I thought it best to write it all out, well to be fair; I want to write about how to write and what to write when you’re struggling with depression. It takes some motivation, perhaps that I just…

In my 41 years of interstellar travel on this great blue spaceship I’ve never experienced anything quite like what’s happening right now. But I’ve traveled about enough to know that my limited time…