Unbundling Google
Ohio State University team creates first closed-world dataset for travel planning
When you book a trip:
25% of your hotel room price gets paid to an OTA
Online Travel Agents like Expedia take that revenue and use it to advertise on Google
20% of Google’s revenue comes from the travel sector
This relationship is so significant that small changes on the Search Engine Results Page have a huge impact, as evinced by Expedia owner Barry Diller’s rant at such a change.
AI that plans a trip is one of the holy grails of product. And now the team at Ohio State University has finally created a benchmark to specifically measure how close we are to creating a Google killer, an AI travel planner.
The benchmark consists of a dataset of training set of 5 queries, with human-annotated solutions, and a test set of 1,000 queries.
The Agent has presumed access to 6 search tools:
City
Flights
Distance
Attraction
Accommodation
Restaurant
And the dataset includes 4 million entries for possible results from the 6 tools. In effect, this is the first closed-world dataset for travel planning which should be large enough to create a planner.
The findings using this dataset and the benchmark were.. interesting:
LLMs are terrible at planning right now - GPT-4 had a success rate of 0.6%, while all other models totally failed
Existing planning strategies fail at this level of complexity.
Google certainly doesn’t seem to need to worry yet… However, in AI, the creation of a good benchmark and dataset is often the first step towards finding the path to a solution… And I‘m sure we’ll be talking about state-of-the-art performance on this benchmark within a year.