Code

Search engine startup Perplexity have introduced a new type of LLM model, which they call an “Online LLM”.

Online LLMs combine the capabilities of an LLM with factual data scraped from the web, that a traditional search engine would provide. The combination is powerful - you get the retrieval power of search combined with the natural language skills of an LLM.

For example, if I wanted to get the phone number of my local starbucks:

$ ppx n get me the phone number for the Bala Cynwyd Starbucks on Montgomery Avenue, PA.
The phone number for the Bala Cynwyd Starbucks on Montgomery Avenue, PA, is (610) 617-9386.

(ppx is a shell script which wraps ch).

Compare this result to Google’s search API, the query “Phone number for Bala Cynwyd Starbucks on Montgomery Avenue, PA” returns 10 results, and the third is from Yelp, which has the phone number in it:

{
  "snippet": "Starbucks · Map · 138 Montgomery Ave. Bala Cynwyd, PA 19004. Directions · (610) 617-9386. Call Now · Full menu · More Info. Hours, Health Score · Known For. Yes.",
  ...
}

In either case though, I’m going to have to parse the response for phone numbers. Can’t I ask the online LLM to do that for me?

$ ppx n get me the phone number for the Bala Cynwyd Starbucks on Montgomery Avenue, PA. Only print the phone number digits do not give me any words.
(610) 617-9386

Now, this is a silly example (perhaps I should have used Google’s Places API) but you hopefully you can see how the online LLM model let’s you bundle a search request with compute.

You might classify this interaction as “search/parse”. For sure there are others; what about searching for two disparate facts and combining them into a single result? (“search/combine”). Or “search/validate”:

$ ppx n 'Is (610) 617-9386 the phone number for the Bala Cynwyd Starbucks on Montgomery Avenue, PA? Only return "yes" if it is the phone number, or "no" otherwise.'
Yes

Perplexity price online LLM requests the same as Google search: 0.5¢ per, which is a little steep if you’re doing a lot of data processing. As competition drives down the cost of LLM compute, I expect this will come down too. In the meantime, despite the AI hype, online LLMs are an innovation that you might actually use.

Tags: online-llm perplexity ch chatgpt openai