The launch of Apple Vision Pro has initiated a flurry of interest in the future of consumer AR. Most predictions about future use cases and increasingly sci-fi capabilities imply that the path towards the "perfect" AR device will arrive over a decade of iterative reductions- smaller devices, lighter batteries, thinner lenses, etc. The platonic ideal of an AR experience seems most likely to emerge by taking today's bulky, uncomfortable hardware and slowly shrinking it into a form factor that is both physically and socially tolerable to the average person. The tech giants creating today's headsets (Apple and Meta, though others will surely follow) have every incentive to continue iterating on the hardware until it reaches that ideal state, at which point we may indeed see widespread consumer adoption.

However, there is an alternative path through which AR could break into the mainstream. While the media has focused almost all of its attention on the heavy-duty devices created by big tech, a parallel branch of AR development has gathered some steam in recent years, driven by startups like XReal, Rokid, and Viture. These products are not bulky headsets but instead lightweight glasses that look like and weigh around the same as a pair of RayBans. Unlike the Vision Pro and Quest series, the devices aren't yet viable "spatial computing platforms" or app ecosystems; a better description might be "AR gadgets". XReal launched its Air 2 Ultra glasses at CES in early 2024, and I've been surprised at how little attention these have garnered relative to the Vision Pro. At $699, they have (admittedly primitive) spatial tracking, can display a 30+ inch screen a foot from the user's eyes, and most importantly, weigh only 80g. Yes, the field of view is a mere 53 degrees to the Vision Pro's ~100. And yes, the spatial tracking leaves much to be desired and the device doesn't have any onboard computing; in many ways, this is an Apple™️ to oranges comparison. But isn't it pretty remarkable that even a limited AR experience like that of the Air 2 Ultra is already possible in the form factor of sunglasses?

I don't think we should be underestimating this less heavy-duty path to widespread AR usage. These startups are also all-in on the race towards the platonic AR experience, albeit with a lot less funding and a lot more to lose. If the consensus bet is correct, and Apple/Meta is the first to build a true consumer AR device, these startups will be dead on arrival. And for what it's worth, that's the outcome probably what I'd bet on. But if the convergent evolution towards widespread AR adoption ends up favoring the "virtual-screen-not-a-spatial-computer" approach favored by the aforementioned startups, this may have some very interesting implications for the strategies that Apple/Meta end up pursuing. They will likely still end up winning the market but may be forced to course correct, and us consumers will end up with different expectations and experiences of AR.

There are a variety of factors that will dictate the outcome of this race. But to understand how some might play out, it's interesting to examine a couple of fundamental questions about what consumers want from the perspective of "Big AR" (Apple/Meta/Big Tech Cos) and "Scrappy Startups" (XReal, Rokid, other AR glasses makers). In some ways, this debate mirrors the thick vs thin client debate in the previous generation of computing technology. Let's begin with some questions where both sides tend to agree, and then delve into the distinct approaches they take.

Question: How valuable is a virtual work station/desk-in-a-headset?

Big AR:

Immensely valuable. The ability to have an arbitrary number of screens powered by the AR device, anywhere in the world, at any time will have huge implications on how we work. One day soon, you'll be able to simply toss on a headset and immerse yourself in productivity on a plane, at a bar, etc. Forgot your computer? No problem, the world has now become your workstation.

Scrappy Startups:

We agree it's valuable, but let's 80-20 rule this thing. Will people want to work in places where they can't even have a computer on their person? Seems pretty rare to me. Most of the value prop here is having multiple screens of multiple sizes in any space. That doesn't require building a modern OS on the device! The virtual screens are just frontends, and we should offload computation to the computers and phones that do them best and don't have to sit on your face for hours. The classic argument about thin clients applies to AR! The glasses are mostly useful as a virtual monitor—let's not overcomplicate it.

Question: Is media consumption the killer AR app?

Big AR:

Yes, this is probably the first significant consumer use case! Imagine sitting back and watching a movie with a 100-foot screen inside the Amazon rain forest- you can do this today in the Vision Pro. Not to mention 3D movies, front-row seats to NBA games, and video games. The more immersive these experiences are, the better, and the more compelling they will be to customers.

Scrappy Startups:

Agree that consuming media is the first killer use case, but 100-foot screens? We've gotten by just fine with 40-inch TVs for decades. Again, is the actual value here the immersion and screen size or just the fact that users can now bring monitor-sized screens onto the Subway or a plane? We'd guess the latter, and so our screens won't be limitlessly gigantic, but they'll be big enough to warrant daily usage. And 3D? C'mon, we tried that in movie theaters 10 years ago, and how many 3D movies are people clamoring to see these days? There's no reason to think it would be any different in a sweaty, bulky headset.

Question: Is a wide field of view essential to mainstream consumer adoption?

Big AR:

Duh. We are in the business of augmenting reality, and the human eye has a very wide FOV. Suppose we don't prioritize expanding FOV as much as possible. In that case, users will constantly be bothered by the dark edges around their screen, won't ever truly feel immersed in the AR world, and will have to constantly turn their heads to see things that should be easily visible in their periphery. Those flaws are non-starters for a breakout consumer product- the 100-degree VP is a first-generation device for a reason. Any FOV much below this has no chance of appealing to the average consumer.

Scrappy Startups:

A wider FOV is nice, but are we sure this is the right goal to be chasing? We won't ever fully replicate the FOV of human eyes, so maybe best to give up on that dream in favor of other pursuits. And unlike your passthrough screens, which leave a black ring around the user's vision, our glasses don't obscure their actual periphery, so the "binoculars effect" is less of a problem. Besides, 90% of the value of an AR experience comes from the content directly in front of the user's eyes. Having a dozen AR screens around you and catching glimpses of them out of the corner of your eye helps preserve visual continuity. But you're going to need to turn your head to engage with any of those screens anyway. Besides, humans aren't going to forget about their other monitor because they can't see it without moving their neck a little bit- we develop object-permanence as infants! Large FOV at the expense of device weight is the wrong tradeoff when the actual utility of an AR experience is considered.

Question: Is powerful on-device compute essential?

Big AR:

To consider an alternative to on-device computing is completely ridiculous. Would the mobile phone have become ubiquitous if it had to be plugged into a computer? Even if we use a wireless connection for some use cases (i.e. mirroring a Macbook), complete reliance on physical proximity to some other source of computation is a non-starter. Without a powerful on-device computer, these headsets won't ever be true standalone tools, won't empower a native app ecosystem, and won't be the 10x improvement on the status quo needed to take them mainstream.

Scrappy Startups:

You haven't thought through what AR is really useful for. Almost nobody will be in a situation where they don't have their computer or phone but want to use their headset to get some work done. Are you seriously betting that humans will have their phones in their pockets less often in the future? On this app ecosystem point—do we anticipate any apps that truly need crazy amounts of native compute besides gaming? Granted, that's a huge market, but that's more of a VR concern. A miniature computer in a pocket (XReal takes this approach with their Beam device) or a phone/laptop connection should be sufficient to power most of the useful apps. Note that computer monitors are not "standalone" products, which doesn't stop them from being immensely useful. Remember, on-device compute == on-your-head-metal-thing, and there is no escaping this fact in the near to medium term. Necks will always be worse at carrying computers than pockets or desks.

Question: Do consumers need futuristic immersive experiences to adopt AR en masse?

Big AR:

We've had AR devices that do many of these "useful" things for over a decade, and none of them have gone mainstream. The reason is obvious: they weren't immersive enough. They didn't blend the real and the virtual well enough. They didn't enable experiences that were mind-blowing enough. AR is already useful- it needs to be much more than just that to trigger an "aha moment" for most consumers. Sure, bulky headsets look weird, but so did the first phones or earbuds. The weirdness and over-the-top factor will make it easier for early adopters to evangelize to the rest of the market.

Scrappy Startups:

Or maybe the reason was that they were too physically uncomfortable? Occam's Razor here: a heavy, sweaty thing stuck to a person's face didn't become a ubiquitous tech product because... it was a heavy, sweaty thing. Sure, devices need to have some level of technical capabilities and wow factor, but if we agree that 80% of the value here is bigger monitors on the plane and NBA games on the subway, maybe we should solve the comfort/social acceptability thing before we build the Holodeck. It may not make for splashy press releases, but utility and ease of use are what drive all consumer hardware trends in the long run. Let's keep the glasses minimalist and light, get the software just good enough to provide a seamless workstation/media consumption tool, and market the hell out of it. And no, looking like a freak with a computer strapped to your head is not how we convince the world to try out a new technology. Wearing a pair of slightly oversized RayBans is so much less of a social faux pas, and guess what? That matters.

Big AR has orders of magnitude more money, and therefore room for error, than the Scrappy Startups. For that reason alone, most of these gadgets will likely never transcend the realm of niche enthusiast markets. However, there are some important lessons to learn when considering what tradeoffs these smaller players are making. In the dialogue above, I find myself agreeing with many of the arguments against the Big AR vision; maybe what the market needs is a back-to-basics approach. We may one day experience AR as a completely immersive world of holograms, where entire new worlds are generated on the fly for our infinite enjoyment. But perhaps the first step towards that world will look less like the V1 Vision Pro and more like an unassuming pair of sunglasses that project a big (but not huge!) computer monitor into our eyeballs.

Why do people keep making new calendar apps? And I'm not talking about CS students trying their hand at a first React.js application (been there, done that). I'm talking about talented, proven entrepreneurs who appear to be violating every startup 101 rule by building commodity software in a crowded category to try and win a fraction of a market dominated by, let's double check, yep, Google and Microsoft! It's no surprise these apps rarely bring something genuinely new to the table. Calendars, and many other kinds of personal productivity software, are just too damn simple to inspire net new software experiences.

The calendar app is the jazz standard of web software—it's exciting to find a fresh take on an old classic, but nobody expects to top the charts with Autumn Leaves. But year after year comes the familiar tune of a buzzy Product Hunt/Twitter launch and a TechCrunch article about a new calendar startup, freshly funded by intelligent and capable VCs, ready to take on the world. What gives?

I know it sounds like I'm being a hater right now. But actually, I love new calendar apps! I've probably gone through five in the past two years (as a paying user, even) and probably will go through five more in the years to come. I don't expect each new app to be noticeably better than its predecessors—it's just really fun to use them all. And there are a growing number of people just like me. This cohort is interesting to consider because here the conventional wisdom about software market dynamics begins to break down and lose its explanatory power. I'll illustrate this by distinguishing two made-up groups, "software users" and "software appreciators".

Software users are normal customers in a consumer or B2B context. They care about what software does more than how it does that thing. They have a problem and look for software that solves it effectively and to the extent that it's possible, invisibly. Software users are the customers that you're supposed to build for. Software appreciators, on the other hand, may not have a "problem", or the problem might already be (mostly) solved. Software users are not merely trying to extract utility from a software experience because, for them, the experience can be an end in itself. When you build delightful UX into your product for software users, it's a subtle, subconscious trigger that improves your retention curves. Software appreciators notice it consciously and choose your product because they hope you can provide those delightful moments in spades.

Appreciators are not quite the same thing as power users, though they usually overlap. Appreciators see the software experience as a work of art more than as providing a function. Personal productivity apps like calendars tend to attract them more than any other category because the old advice that "the best notes app is the one you use" is true! With a basic solved-problem functionality as table stakes, it's the cherry on top that sells the milkshake. These products exist in a market where Hotelling's Law has long since run its course, and they can no longer differentiate on the features that make them valuable as consumer utilities. So they develop diverging feature sets, generally with small UI/UX changes, that appeal to the mercurial appetite of the software appreciator.

This is a cyclical process—the apps add new capabilities and easter eggs right up to the point that a new competitor emerges with a minimalist landing page and a "back to basics" product ethos. To be clear, even in these markets, software appreciators are a tiny subset of software users. But they form a vocal minority of true evangelists, and their obsession with tiny details that no normal customer would notice can bubble up into momentum that gets the app reviewed by the Ali Abdals of the world and thrust into the mainstream. That's the bet that everyone building these products is making; build for software appreciators and the users will come.

So why is this phenomenon interesting, beyond a quirk of some niche software markets? Because there is a real chance this strategy will become increasingly important as a result of two tailwinds: Internet Nativity and Generative AI. Being a software appreciator, like being an art critic, requires immersion. To understand why the keyboard shortcut in xyz calendar app is so awesome, you have to have tried out the other 10 apps that don't have it. And so it is with all other software products. An increasing percentage of the Internet software market is comprised of people who have spent their entire conscious lives using software. Today's iPad kids are tomorrow's tech employees—these people will have taste in software that many of us haven't the palate for. They will have opinions about niche details the average user ignores today because they experience software with higher fidelity. The internet isn't a second language for them; they will expect more from it, even at work.

Generative AI, on the other hand, is already changing the way we build products. The tldr is that it makes software a whole lot easier to make, and make quickly. It will be harder and harder to build a utility feature moat if your competitors can replicate it with a junior engineer and GPT-N in the afternoon. So products will be forced to differentiate on the dimension that can't be copied so credibly: style. GenAI may also make it easier to switch between providers (because LLMs make arbitrary data transformations trivial), so the future of all kinds of software products, not just calendars, maybe more similar to the fashion world than the internet boom. Even big companies with huge market caps may tend towards constantly cycling styles, remixing retro themes, and making very opinionated design choices.

If text is truly the universal interface, I think we are rapidly approaching a day when most apps can do most things. You'll probably be able to ask your CRM which part of the codebase a qualified lead's request would most directly impact, or do cohort analysis from your IDE. So the best teams will build for the software appreciators, who grow in number each year.

This post was inspired by the launch of Amie, a new calendar app. The landing page has some flashy and unnecessary animations. It's got bubbly fonts and a friendly feel, and makes it 5% easier to schedule a meeting. It integrates with the standards tools you'd expect it to. As far as I can tell, I replicate all of its features directly in Google with a single extra click maximum. But of course, I tried it, absolutely loved it, and immediately downloaded the desktop app and dropped the iOS widget into my home screen.

If you've been employed as a programmer for fewer than 5 years, you stand to gain a lot from aggressively adopting AI coding tools. Despite being in the best position to benefit professionally from language model outputs, the vast majority of engineers I talk to are still reluctant to go truly all-in on using them. I get it—routines are efficient, and compared with most knowledge workers we tend to be unusually attached to our current workflows. But don't make the mistake of seeing AI tool adoption as something akin to learning a new framework or language, where to succeed is to merely replicate your current efficacy in some new domain. Holding actual effort constant, these tools, when used correctly, can make you significantly more effective. Or they can maintain your current productivity while allowing you more time to fuck around at work... I recommend striving for the former approach!

There are plenty of tutorials and "5 ways to use ChatGPT!" tweets you can find for specific instructions and tools. I won't waste time with the nitty gritty, none of this is rocket science. What I will do is offer some advice about how to approach the meta-game of applying AI to your engineering tasks.

Some background on why I think my thoughts here are relevant: I started programming professionally (full stack at a startup) about a year before Github Copilot launched in beta. That was enough time to get a sense of the tasks that generally end up on an IC's plate, but certainly not enough time to master the requisite skillsets. When I began using Copilot and ChatGPT, I was at a point where my rate of skill acquisition was accelerating; I hadn't yet hit a plateau. So using the AI tools became part of the process by which I was improving generally as a SWE, and I was therefore in a good position to figure out which tools and approaches were really useful. Since then I've become an avid Cursor and Copilot Chat user as well, and have tried most of the other coding assistants on the market.

The most important thing to understand about AI tools is that you should conceptualize them as a human pair programmer (albeit a not very intelligent one). The tools are kind of like a coworker when you drag them into a spontaneous Slack huddle—an entity with a lot of domain-specific knowledge, but usually completely blind to the specific problem you have. And like in this real-world situation, you must put a lot of effort into "catching up" the LLM as quickly and accurately as possible. Let's say you have some coding task "X" that you hope to accomplish with AI. At least 50% of the total mental effort you exert on X should be spent figuring out the clearest way to communicate that problem. It is tempting to just break X into, say, five steps, and immediately start solving step one with AI assistance. This can be a valid approach in some situations, but you'll often find that by step three or four, your failure to communicate the entire problem has caused the LLM to miss the mark by a wider and wider amount. I make this mistake all the time, and the source is always just plain laziness. Always try your best to capture the entirety essence of a problem in your initial prompt.

The problem statement of X need not and should not scale linearly with the size of X. You can almost always communicate the gist of the problem with toy examples or simple hypotheticals. And you can use AI to help with this! So instead of breaking down X into 10 problem statements and tackling each one, start with a few smaller prompts asking the LLM to (for example) rewrite your class in a condensed version with implementation details removed or summarize long console error outputs into shorter phrases. The smaller the context window, the more information about your problem that is packed into each token, and the more likely it is that your LLM will produce a useful result. Hardly a revelation, but as context windows have grown with each model release it's become very tempting to start ignoring this fact. Use prompts to build better prompts; it's often worth the effort.

Shorter contexts are also more readable. Readability is important! Not for the LLM per se, but for you the developer. I've made the mistake of just dumping text into a context window and hitting send, only to find that three questions later I've totally forgotten/misunderstood my problem. Do not underestimate an LLM's ability to trick you into thinking you're getting closer to the answer. You could argue it's literally designed to do this. If you write clear problem statements, you can reference them as the conversation continues, and if you decide the LLM has strayed too far from the desired path, simply start the conversation over from the initial statement.

That's another important tip! Far too few developers utilize the "restart from earlier in the conversation" features in these tools. Just think of the LLM as a colleague with a bad habit of digressing from the topic at hand; sometimes you need to hit them with the classic tech refrain, "Let's circle back". In the context of an LLM tool, this usually means restarting the chat from a prior message. This is often more productive than simply asking your question again in a new way, or telling it to try again, etc. While that can work in some situations, remember that you risk polluting the context with useless tokens. The important information may end up getting lost in the middle; this is statistically more likely the longer the conversation goes on. When in doubt, try rewriting your original problem statement instead of continuing a frustrating conversation with a confused LLM.

LLMs in early 2024 rarely produce useful code output in chunks larger than 30-40 lines (that is, "problem-solving" code—they can do much more if they are simply copying input or generating boilerplate). Strive to develop a sixth sense about when an LLM output is probably too long to be trustworthy. And of course, this varies depending on the language/framework/task. The LLM is a lot more likely to produce useful large chunks when implementing a common React.js pattern than something very idiosyncratic. Develop an intuition about this too. When you start out with these tools, I think it's worth asking LLMs to do things more often than you actually need them because it helps you develop that instinct for their limitations more quickly. At this point, I rarely expect LLM-generated code to work and discover that it doesn't.

But don't ignore my parenthetical in the prior paragraph! LLMs are extremely useful when for copying code or generating boilerplate. You should basically never write boilerplate files anymore. Using something like Cursor chat, just say "Make a new boilerplate file called my_file.ts structured in the same way as @my_other_file.ts". Build a mental toolkit of these short script-like instructions that work well in your codebase. Just be sure to understand when you're asking the LLM to actually solve a problem and when you're asking it to do some worker-bee task for you and adjust your expectations accordingly. The line between these is fuzzy, but not fuzzy enough to make the mental categorization impractical.

Not all problems are suitable for LLM assistance. The most ideal are those that have a straightforward “shape” but require you to hold more information in your head at once than is comfortable. Here's a decent heuristic: to solve this problem unassisted, would it be helpful to write something down (aside from the code itself)? If yes, it's often worth taking the time to communicate it succinctly to an AI tool via chat. There is a different threshold of complexity for different people and different tasks. Figure out where yours is and avoid the mistake of asking questions that don't meet that threshold. If you ask the wrong questions, a chat-based AI tool will probably slow you down. It may seem more efficient to have an LLM write your code, but often, correcting LLM mistakes can take longer than if you had written the code yourself. Knowing when to use a chat window or opt for simpler autocomplete tools like Copilot is a meta-skill that underlies all this advice.

A lot of people seem to think that you should never blindly trust an LLM’s output. I get where they're coming from, but this is an objectively bad heuristic. There are plenty of times when I feel 100% confident trusting it without modification, but that's because I've developed an intuition about its limitations in the context of my codebase. This takes at least a few weeks for a given codebase, assuming you work a normal SWE job. And yes, if you can't roughly explain what generated code does then obviously don't check it in, and if it has security implications then additional scrutiny is warranted. My point is, in many situations, reviewing the output from these models requires no more attention to detail than you would normally give to a junior engineer's PR. It's even possible that doing so may one day be considered a bad habit—as the models improve, they become more and more trustworthy, and the compulsive need to double-check them in every scenario will end up slowing you down.

Finally, try your best to refrain from cursing out the LLMs (note to self...). They are doing their very best and it's wise to establish a good relationship while you're still the superior programmer!