Perplexity beats Gemini Pro
Over the last couple of days I was working on a relatively complex software integration project for image generation with stable diffusion. I had dozens of different pieces from multiple websites to download, configure, integrate, and debug. None of it was particularly well documented. In the past, I would have read what documentation existed, most of which was out of date, experimented a lot, and asked a number of questions on Reddit or other forums. This time, instead of bothering a lot of people online, I decided to use LLMs and chatbots to assist me and explain to me in detail what I needed to do when and how to accomplish my task. The results weren’t pretty.
I started with Gemini Pro, which I have a subscription to as part of Google One. I began with Flash and eventually switched to the thinking model when the Flash model gave results that were simply too poor to be believed. Over two days, Gemini hallucinated routinely, sent me down many wrong paths, suggested things I specifically told it not to suggest, and in general didn’t solve the problem. The next day I decided to try Perplexity, which I also have a subscription to. Perplexity also hallucinated, and it wasn’t perfect, but it did a better job than Gemini of following my instructions and answering the questions I asked instead of going off on tangents I was trying to steer it away from. Most importantly, it did finally lead to an answer and a working system.
This wasn’t a particularly scientific test. When I started working witherplexity, I had already established a number of things from my sessions with Gemini. So there were things I tried with Gemini I didn’t even try with Perplexity. Nonetheless, perplexity got me 90% of the way to a fully working solution when Gemini was still struggling to try to figure out where various files were laid out in my particular file system.
The biggest problem I noticed with Perplexity, and perhaps to a lesser extent with Gemini (lesser only because because Gemini created bigger problems that Perplexity simply didn’t) was a tendency to confidently refer to features in the UI I was working with that simply didn’t exist. Possibly both Gemini and Perplexity had been trained on much older versions of the software I was trying to configure. Nonetheless, it was quite frustrating that both systems ultimately resulted to guessing what might be there instead of looking and seeing what was actually there, either on my own Mac or in the current documentation and software available on the web.
Both systems also had a really annoying habit of giving me links to download software and libraries and weights that were 404. Gemini was even worse here because it would not give me direct links to anything. Every link it created was always redirected through Google search. So, for example, it couldn’t link to http://www.example.com. It would only link to https://www.google.com/search?q=http%253A%252F%252Fwww.example.com. I wasted a lot of time trying to convince Gemini to give me direct links that were not wrapped in a Google search result, but apparently the front end does that and the Gemini LLM doesn’t have control over it.
All that said, this was still a clear win for Perplexity. Perplexity solved my problem and got me to a working system when Gemini couldn’t.
