Search for a command to run...
Hands-on technology leader with 10+ years building scalable, mission-critical systems at Goldman Sachs, Brevan Howard and fast-growing fintechs. Expert in cloud-native architectures, distributed data pipelines and high-throughput systems; experienced in migrating legacy platforms and designing AI-enabled services. Proven track record delivering reliable platforms that process millions of transactions daily.
No comments yet. Be the first to comment.
It’s an Exoskeleton, and That Changes Everything Most people are thinking about AI the wrong way. They imagine it as a coworker:Something you assign tasks to… wait for… and hope it delivers. Sounds re
I didn’t wake up one day and decide, “AI will change everything.”My journey into AI was slower, messier, and honestly… a bit reluctant at first. If you’re a builder, engineer, or curious technologist,
Imagine a programming language designed not just for humans but also for AI.Not adapted for AI. Not retrofitted. Built from scratch so AI can read and write it easily. That’s exactly what NanoLang is. It’s a tiny, experimental language created by vet...
Most PostgreSQL optimization guides feel like laundry lists of settings and indexes. But real performance gains often come from clever ideas, not just the usual tricks. Let’s unpack a few of those ideas in ways that actually make sense for you. These...
Remember when we defeated pop-up ads? Yeah, they're back. And this time, nobody's fighting them. The Good Old Days (Of Terrible Ads) Back around 2000, the internet was a warzone. Visit any website and BAM!!! random windows would explode onto your scr...
In a previous post I explained about SSTable and their role in the persistent storage of data in Cassandra.
Suppose that following query is received on a node (news_id is primary key):
SELECT * FROM tblNewsItems WHERE news_id = '8bacfe891fa89bfab98d9e99f9a9';
How can the node determine if this specific row exists in it’s local data? As we know we have in-memory tables (mem-table) but they are used as a caching mechanism and not the real storage. So any data lookup in these in-memory storages won’t be enough as they don’t have the whole range of data.
Another solution is to read the SSTable and check if the table contains the key that we want. Even in case of using effective search algorithms like binary search, this will be too expensive because it involves a lot of disk reads which may not be efficient enough for a system with a lot of data and a lot of read queries.
The solution is to use Bloom Filter data structure. This is an in-memory, probabilistic data structure which can help you find if an element is member of a set or no? Here the set is the Cassandra table (which is actually a set of rows) and the element is the primary key we are looking for. So we have these features:
A bloom filter is a bit-map containing n bits and a set of hash functions which are expected to be independant. Let’s assume we have a bloom filter with 100 bits and 3 hash functions (f1, f2 and f3). In practice the size of bloom filter will be much larger (MB or GBs). Each of has functions will receive an arbitrary input (row primary key) and output a number between 1 to n. You can easily achieve this output by using modulo operator and common hash functions.
There are two algorithms for insertion of data into a bloom filter and checking whether something exists in the set or no. These alrogithms are pretty simple and straightforward. To insert data x into bloom filter:
x. So we will have a=f1(x), b=f2(x) and c=f3(x).a,b and c in the bloom filter to one (Note that they may have already been set to one but this doesn’t matter).To check if data x exists in the bloom filter:
d=f1(y), e=f2(y), f=f3(y).d, e, and f in the bit-map of bloom filter. Are they all set to one? If so, the row probably exists in the set. If even one of those bits is zero, means that row definitely does not exist in the set.False positive for a bloom filter means cases where the filter indicates that a row exists in the table but it was not. Remmber that this is a probabilistic data structure so these cases may happen.
If there is a false positive, we won’t know about it until we scan the SSTable. So generally Cassandra will scan the SSTable looking for a specific row, if the bloom filter indicates the row exists in the table. If after scan completion, row is not found, this will be recorded as a false positive.
You can run ‘nodetool cfstats‘ command in a Cassandra node to view a lot of statistics about the node. One of those stats is for bloom filters which shows you the memory consumed by bloom filter and number of false positives.