Semantic search is transforming how we find products

An e-commerce platform lives or dies by its ability to connect customers with the products they need. For decades, this task has been largely accomplished by the humble search bar – input a keyword or two, and be rewarded with a list of items with matching terms.

But the world of tech never stands still – especially in the age of AI. As a company with e-commerce close to its heart, Rakuten’s talented engineers have embarked on a mission to take search to the next level. Lee Xiong leads the deep learning initiatives at Rakuten Group, aiming to infuse Rakuten’s search and recommendation technology with the latest AI technology. Xiong is a pioneer in applying deep learning technology at a large scale and had worked on Bing at Microsoft before joining Rakuten. Today, Xiong’s work span an even grander scope, covering a broad portfolio of Rakuten products across multiple industries.

“Rakuten Group is a fantastic playground for engineers as it has one of the world’s most diverse portfolio of products, rich and high-quality data as well as ample computing power. Engineers have the opportunity to amplify their impact by applying state-of-art technologies to create value for our customers in Japan and worldwide.”

-Ting Cai, Chief Data Officer of Rakuten Group, Inc.

What’s new in search?

Keyword matching has long been a staple of any website or app with a search bar, but it is defined by its limitations. A user searching for a specific keyword will typically only be presented with products that contain that exact keyword, leaving out words that might be synonyms or slightly different phrasing.

Xiong’s team is looking to improve this by adopting something called semantic search, which does exactly as the name suggests: It conducts searches based not just on the words searched, but what they mean.

“Traditionally, users have needed to synthesize their query into a few keywords, maybe [T-shirt, cotton]. Now, they can search for a [cotton T-shirt for kids with animal patterns] – complicated queries that go beyond three or four words – and they will still get relevant results.”

The team began trialing the tech at Rakuten’s apparel e-commerce platform, Rakuten Fashion. With its diverse array of fashion products, the site proved the perfect test ground for revolutionizing search.

“This is a disruptive technology that advances the paradigm of search, so we need to be careful not to surprise merchants who might not be used to it,” Xiong says. “We decided to start with Rakuten Fashion, and we received a lot of support from them. They were very brave in trying it out!”

Xiong gave another example of a search query in Japanese along the lines of [casual socks that aren’t too thick made from comfortable cotton]. On Rakuten Fashion, the query produces a selection of closely matching items while a leading competitor’s site returns no results for the same query.

“The only limit is whether or not the platform carries it in their inventory with an accurate description,” he says. “Phrases that on other platforms will land you zero results; this system will always give you something.”

Semantic search: How does it work?

Xiong explains that the system is much more than a conversion layer that translates complex queries into simple keywords.

“The secret behind it is an academic term called dense retrieval,” he explains. “On a high level, what it does is it vectorizes natural language so that it can represent a search query or an inventory item in a vector.”

In a machine learning context, vectors are simply a way to represent a word or an item as a list of numerical values. Words or items with similar meaning will be close in the vector space. “It’s like neurons firing signals – a kind of representation that’s understandable by deep learning models.”

When input data is ‘vectorized’ in this way, it can be processed mathematically, and machine learning can take place.

“So these vectors have hundreds of dimensions, including all kinds of semantic meaning, ranging from categories and colors to age and more,” Xiong continues. “The magical part is that we don’t have to manually specify all these dimensions. The deep learning model identifies the most important dimensions by itself.”

Because everything is in a vector space, any search query will always be able to find the closest match in a given inventory. “It eliminates zero-hit scenarios.”

Vectorizing Rakuten’s high-quality e-commerce data

“For any embedding of the vectorization models, the key is to find pairs,” Xiong reveals. “For example, given [A], we want [B], and not [C]. So we just need to find [A], [B] and [C].”

Xiong’s team can utilize Rakuten’s wealth of search and purchase data to find these pairs and train the model.

“In the context of search, [A] would be the query. Given this query, which item does the user want [B]? Which item do they not want [C]? Which item the user wants is easy, if you know what item the user ends up purchasing from among the search results. “

Now, the model has worked out [A] and [B]. “Which item this user doesn’t want –  you can, for example, start with randomly picking an item from the inventory that the user didn’t purchase,” he explains. “With that information, the model will learn: Given this, I should be looking for this type of item.”

A major benefit of using deep learning is that this entire process happens autonomously.

“With classic search technologies, you would need to find out all the terms that match and write many rules for ranking. But with deep learning, you throw all of these problems to the model, and it figures out how it should match all the terms by itself. Sometimes the user might have a typo or a different way of describing an item, but the model will still understand.”

The key to making this process a success lies in the quality of the training data.

“One of the biggest challenges for any deep learning work is creating very high-quality training data for the model that we want to build,” he reveals. “In this case, we need to create a set of very high-quality Japanese user e-commerce purchase behavior data, together with catalog data containing item details such as the title, description, reviews, price and more.”

It’s here that access to Rakuten’s powerful e-commerce data reveals its true potential: “While it’s challenging, I actually think Rakuten is in the best position to solve that challenge. Other companies will find it even more difficult, if not impossible.”

Letting the results speak for themselves

Initial results from the team’s experiments show promise: Rakuten Fashion enjoyed a 5% jump in GMS from search.

“It’s because we have a deeper understanding of both our users and our inventory,” Xiong explains. “Now that our results have become a lot more relevant to what users want, they are able to find what they need and make the order.”

Another important area of e-commerce that Xiong and his fellow research scientists are also working on are product recommendations. The team has shipped deep learning-driven recommendations on Rakuma, Rakuten’s second-hand goods marketplace.

“Because we can vectorize all the inventory, we can find similar inventory items. We can also take into account the purchase or browsing history of the user and represent the user as a vector, and find which items are most likely to be purchased next using dense retrieval.”

These experiments have also yielded impressive results, with Rakuma seeing a 30% improvement in conversion through the home page user feeds.

Xiong is excited to see where the technology could be applied next.

“Similar technology can be shared between different services, so we can develop once, and use it many times. It’s pretty economical in that sense,” he reasons. “I think many Rakuten services will benefit, as long as they have some kind of search or recommendation function. Rakuten Ichiba, Rakuten Books, Taiwan Rakuten Ichiba, and even Rakuten Music – we can support them.”

Pioneers of Japanese AI

One concern often raised in the field of AI is the dominance of English-based training data, and how other languages could be left behind. Xiong’s work bucks that trend: “Here, because I get most of the data in Japanese, the model actually works much better in Japanese.”

But the team’s ambitions don’t stop with Japanese market.

“Our next goal is to make the model multilingual. We want to support searches so that even if our inventory is fully in Japanese, you can still search in English and other languages,” he says. “I don’t think a translation layer for search is necessary. The multilingual vectors are the most scalable representation of information.”

Competing on the world stage

Xiong believes that Rakuten has the right ingredients to drive AI tech on a global level.

“First of all, you need a strong team of people who know how to develop this technology,” Xiong explains. “Subsequently, the training data is key – and Rakuten really has an advantage in that area because of our extensive e-commerce data – we could construct the most comprehensive e-commerce Japanese training data for any such deep learning model. I don’t think even other major tech companies could build such a strong e-commerce dense retrieval search engine as Rakuten, because they lack the kind of training data that Rakuten has.”

Xiong isn’t shy about his team’s ambitions.

“The goal of my team is to deploy cutting-edge deep learning technology for search, recommendations and other applications, with the aspiration of being on par with major tech giants like Google, Microsoft, and Meta,” Xiong declares. “We have already achieved impressive results and are actively transitioning Rakuten into a leading tech company.”