Embeddings

AI is paid by the word. That’s not a metaphor — tokens are literally what you pay for. That’s why AI loves to be verbose, to generate more “tokens”… it’s a feature, not a bug. This is literally how they reason, and this is how most AI is monetized, what you pay. “Vibe coding”, “Thinking”, “planning”, “agents” … layers upon layers of less and less visible internal narrative you paid. This is only getting worse as the “price per token” (what the “pricing page” shows) gets cheaper and we follow the Jevons paradox. There isn’t much attention to tokens used per outcome. ...

In the field of genetic ancestry, services like 23andMe enable us to trace our roots back 100,000 years. However, the recent ancestry — the closer generations to us — are paradoxically harder to trace. Why? This is in part due to advent of increasingly common, fast, and cheap travel: horses, ships, trains, cars, planes, … It basically mixes populations in much deeper, complex, dynamic ways. The technological revolution made genetic ancestry possible, but also complicates the assumption that one can easily relate location with DNA. Similarly, today, a comparable shift is taking place in the digital world. As I explain below, we can no longer assume that any digital record is a factual representation of the present or the past. This was the case before. It is the end of presumption of digital veracity. ...

AI dominates every digital space. More specifically OpenAI with ChatGPT and GPT4, clearly outperforming [at least in public perception] usual suspects like Google who claimed to be “AI first”, since 2016. The irony is that OpenAI uses Transformers for its star product ChatGPT and friends, which is a Google invention from 2017. There are many reports that Google sounded the alarm to catch up, and yesterdays Google IO shows they are all-in. But a comparison between OpenAI and Google strategies, in my opinion, reveals starkly contrasting strategies: OpenAI goes loudly and boldly all in on generative text, and Google (more silently) on embeddings. ...