2026-01-18
31 min readUnderstanding Embeddings: Part 4 - The Limits of Static Embeddings
Why one vector per word cannot capture meaning—and what this limitation reveals about the nature of language understanding.
EmbeddingsPolysemy
Hi, I'm Anirudh
I am a software engineer who builds distributed systems and backend infrastructure. Most of my work involves figuring out why things break, how to make them faster, and how to keep them running when parts inevitably fail.
I spend a lot of time on the "boring" stuff that turns out to matter: consistency trade-offs, failure modes, and understanding what's actually happening under the hood. Lately, I have been exploring AI engineering: how embeddings work, how to build retrieval systems, and how software engineers can work effectively with LLMs without needing a PhD in machine learning.
I write here to organize my own thinking and share what I learn. If you like understanding how things work rather than just using them, you might find something useful.
Knowledge Domains
Technical areas I explore through writing, building, and hands-on experimentation.
🌐
Distributed Systems
Consensus algorithms, consistency models, fault tolerance, and distributed state management.
🗄️
Database Internals
Storage engines, query optimization, indexing strategies, and transaction management.
🏗️
System Design
Scalability patterns, architectural trade-offs, and designing for high availability.
⚡
Performance Engineering
Profiling, optimization techniques, bottleneck analysis, and latency reduction.
☁️
Infrastructure
Kubernetes orchestration, containerization, cloud architecture, and IaC patterns.
⚙️
Backend Development
RESTful APIs, microservices architecture, data pipelines, and event-driven systems.
🤖
AI Engineering
RAG systems, vector databases, embedding models, and production ML infrastructure.
🧠
Machine Learning
Model training, feature engineering, ML pipelines, and deployment strategies.
🧠
My Engineering Philosophy
🎯 First Principles
Break down complex problems to fundamental truths. Build from the ground up, not assumptions.
🔄 Reproducible Systems
Every system should behave predictably. Infrastructure as code, deterministic builds, comprehensive testing.
🧩 Systems Thinking
Understand component interactions. Optimize for the whole, consider second-order effects.
Latest Technical Insights
Deep dives into distributed systems, performance optimization, and architectural patterns
2026-01-17
43 min readUnderstanding Embeddings: Part 3 - Global Statistics and GloVe
From local context windows to global co-occurrence matrices—understanding how GloVe unifies prediction and counting methods.
EmbeddingsGloVe
2025-12-26
40 min readUnderstanding Embeddings: Part 2 - Learning Meaning from Context
Where do embeddings actually come from? Understanding Word2Vec, Skip-Gram, CBOW, and the learning dynamics that create semantic geometry.
EmbeddingsWord2Vec
Join my newsletter - The Main Thread
Get insights on building scalable systems, performance optimization, and architectural patterns. Be among the first to receive deep technical content as I build this newsletter.
Join The Main Thread - insights on distributed systems and backend engineering.
Powered by beehiiv
Powered by beehiiv
Weekly insights
Real examples
No spam