Commit f66016af authored by  Joel  Oksanen's avatar Joel Oksanen

Finished report

parent 1d289acc
This diff is collapsed.
......@@ -2,6 +2,69 @@
\chapter{Extracted ontologies}
\label{sec:ontology_appendix}
\vfill
\begin{figure}[h]
\centering
\footnotesize
\begin{subfigure}{\textwidth}
\centering
\begin{forest}
for tree = {draw, l=2cm}
[{\textbf{video game}, game, product, title}
[graphics]
[gameplay]
[controls]
[price]
[storyline]
[sound
[music, l=1.5cm]
]
]
\end{forest}
\caption{Extracted ontology}
\label{fig:full_game_ontology}
\end{subfigure}
\vspace{2cm}
\begin{subfigure}{\textwidth}
\centering
\begin{forest}
for tree = {draw, l=2cm}
[{\textbf{video game}, computer game, virtual reality}]
\end{forest}
\caption{WordNet ontology}
\label{fig:wordnet_game_ontology}
\end{subfigure}
\vspace{2cm}
\begin{subfigure}{\textwidth}
\centering
\begin{forest}
for tree = {draw, l=1.5cm}
[video game
[win or lose]
[effect of make you laugh]
[winner]
[fun]
[win]
]
\end{forest}
\caption{COMeT ontology}
\label{fig:comet_game_ontology}
\end{subfigure}
\vspace{0.5cm}
\caption{Full ontologies for \textit{video game}}
\end{figure}
\vfill
\begin{sidewaysfigure}[h]
\centering
\footnotesize
......@@ -331,60 +394,3 @@
\caption{Full ontologies for \textit{stand mixer} (*the term \textit{stand mixer} returned no results in WordNet, so \textit{mixer} was used instead)}
\end{sidewaysfigure}
\begin{sidewaysfigure}[h]
\centering
\footnotesize
\begin{subfigure}{\textheight}
\centering
\begin{forest}
for tree = {draw, l=2cm}
[{\textbf{video game}, game, product, title}
[graphics]
[gameplay]
[controls]
[price]
[storyline]
[sound
[music, l=1.5cm]
]
]
\end{forest}
\caption{Extracted ontology}
\label{fig:full_game_ontology}
\end{subfigure}
\vspace{1cm}
\begin{subfigure}{\textheight}
\centering
\begin{forest}
for tree = {draw, l=2cm}
[{\textbf{video game}, computer game, virtual reality}]
\end{forest}
\caption{WordNet ontology}
\label{fig:wordnet_game_ontology}
\end{subfigure}
\vspace{1cm}
\begin{subfigure}{\textheight}
\centering
\begin{forest}
for tree = {draw, l=1.5cm}
[video game
[win or lose]
[effect of make you laugh]
[winner]
[fun]
[win]
]
\end{forest}
\caption{COMeT ontology}
\label{fig:comet_game_ontology}
\end{subfigure}
\caption{Full ontologies for \textit{video game}}
\end{sidewaysfigure}
\ No newline at end of file
\chapter{Conclusion}
In this project, we have extended the work of Cocarascu et al.\ \cite{RefWorks:doc:5e08939de4b0912a82c3d46c} by proposing a general-domain conversational review explanation system, ADA-X, and implementing it as a server-client application with a user-facing messaging interface. The generality of ADA-X is provided by a novel domain-independent ontology extraction method using the state-of-the-art language model BERT, which we have shown to obtain ontologies with an F1-score multiple times higher than the popular general knowledge networks of ConceptNet and WordNet. Furthermore, we have implemented a state-of-the-art feature-based sentiment analysis method, which we have shown to perform with near-optimal accuracy when applied in the general-domain despite being trained on domain-specific data.
In this project, we have extended the work of Cocarascu et al.\ \cite{RefWorks:doc:5e08939de4b0912a82c3d46c} by proposing a general-domain conversational review explanation system, ADA-X, and implementing it as a server-client application with a user-facing messaging interface. The generality of ADA-X is provided by a novel domain-independent ontology extraction method using the state-of-the-art language model BERT, which we have shown to obtain ontologies with an F1-score multiple times higher than the state-of-the-art general knowledge networks of ConceptNet and WordNet. Furthermore, we have implemented a state-of-the-art feature-based sentiment analysis method, which we have shown to perform with near-optimal accuracy when applied in the general-domain despite being trained on domain-specific data.
ADA-X has been shown to be capable of providing dialogical feature-based explanations for any Amazon product, and its methodology could be extended to any review domain with sufficient amounts of data. Our user experiments have shown that the explanations provided by ADA-X improve the user experience of browsing product reviews, and that they can be communicated via different user interfaces for different purposes.
......@@ -11,7 +11,7 @@ The focus of this project has been largely theoretical, as a large part of the p
\subsection{Extending the conversational aspect}
The current argumentation dialogue of ADA-X is quite simple, relying on just two unique query templates in which arguments are embedded. One way to make the conversations with ADA-X more varied would be to increase the number of different queries in the dialogue. For example, one could easily extend the dialogue by adding the query \textit{What were the most polarising features of \underline{\quad}?}
The current argumentation dialogue of ADA-X is quite simple, relying on just two unique query templates in which arguments are embedded. One way to make the conversations with ADA-X more varied would be to increase the number of different queries in the dialogue. For example, one could easily extend the dialogue by adding the query \textit{"What were the most polarising features of \underline{\quad}?"}
Furthermore, the phrase examples given by ADA could be more effective if we could highlight the portion of the text where the sentiment is expressed. For example, in the relatively long review phrase
\begin{center}
......@@ -21,11 +21,11 @@ it would be helpful if we could highlight \textit{"this brand makes beautiful an
\subsection{Recommender system}
Perhaps the most practical application of ADA-X would be within a recommender system, where it could provide the user with feature-based explanations for why a product was recommended to them. It has been shown that explanations can help improve the effectiveness of a recommender system \cite{RefWorks:doc:5e2f3970e4b0241a7d69e2a4}.
Perhaps the most practical application of ADA-X would be within a recommender system, where it could provide the user with feature-based explanations for why a product was recommended to them. It has been shown that such explanations can help improve the effectiveness of a recommender system \cite{RefWorks:doc:5e2f3970e4b0241a7d69e2a4}.
But rather than just basing its explanations on the features of products, an entire recommendation system could be built on the feature-based representations that ADA-X is capable of extracting. To this day, Amazon's recommendation system is still largely based on \textit{Item-to-Item Collaborative Filtering} \cite{RefWorks:doc:5ee67953e4b0e3a229e0b5f8}, a method invented at Amazon in 1998. This method is based on correlations between products in terms of customers' purchase histories, and although focusing on such a singular view of products used to make sense in terms of computational costs, high-performance computing has come far since twenty years ago.
The focus on products as singular entities ignores any feature-level preferences of users: for example, one group of people might be more interested in the \textit{design} of items, while a more practically-minded group might look for \textit{durability}. Furthermore, unlike products, many features are relevant across several product categories; a feature-based recommendation system could take advantage of such inter-categorical connections. For these reasons, we propose that a \textit{feature-to-feature} approach to collaborative filtering based on the feature-based product representations of ADA-X could improve the accuracy of Amazon's recommender system.
The focus on products as singular entities ignores any feature-level preferences of users: for example, one group of people might be more interested in the \textit{design} of items, while a more practically-minded group might look for \textit{durability}. Furthermore, unlike products, many features are relevant across several product categories; a feature-based recommendation system could take advantage of such inter-categorical connections. For these reasons, we propose that a \textit{feature-to-feature} approach to collaborative filtering using the feature-based product representations that ADA-X is capable of extracting could improve the accuracy of Amazon's recommender system.
......@@ -5,7 +5,7 @@ In this chapter, we will first discuss the motivations behind the project and th
\section{Motivations}
People spend an ever growing share of their earnings online, from purchasing daily necessities on e-commerce sites such as Amazon to streaming movies on services such as Netflix. As the market shifts online, people's purchase decisions are increasingly based on product reviews either accompanying the products on e-commerce sites, or on specialised review websites, such as Rotten Tomatoes\footnote{https://www.rottentomatoes.com/} for movies. These reviews can be written by fellow consumers who have purchased the product or by professional critics, such as in the latter example, but what unites most online review platforms is the massive number of individual reviews: a single wrist watch can have more than 20,000 reviews on Amazon\footnote{https://www.amazon.co.uk/Casio-Collection-Unisex-Adults-F-91W-1YER/dp/B000J34HN4}. As people cannot possibly go through all of the individual reviews, purchase decisions are often based on various kinds of review aggregations. The presentation of a review aggregation must be concise and intuitive in order to be effective, but it must also retain some nuances of the original reviews, so that consumers can understand \textit{why} a product is considered good or bad and if the arguments align with their individual preferences. \par
Explanations of review aggregations are also relevant to e-commerce site recommender systems, as the recommendations are often largely based on existing consumer reviews. It has been shown that explanations can help to improve the overall acceptance of a recommender system \cite{RefWorks:doc:5e2f3970e4b0241a7d69e2a4}, and therefore insightful explanations of review aggregations are important to their effectiveness.
Explanations of review aggregations are also relevant to e-commerce site recommender systems, as their recommendations are often based on existing consumer reviews. It has been shown that explanations can help to improve the overall acceptance of a recommender system \cite{RefWorks:doc:5e2f3970e4b0241a7d69e2a4}, and therefore insightful explanations of review aggregations are important to their effectiveness.
\section{Objectives}
......@@ -27,19 +27,19 @@ Some systems have already been proposed to improve explanations for review aggre
In order to participate in such dialog, ADA has to have knowledge about the \textit{ontology} of the product (i.e. \textit{acting} is a sub-category of \textit{film} and \textit{Meryl Streep} is a sub-category of \textit{acting}), as well as knowledge about the reviewers' sentiments towards the various aspects of the film. ADA obtains some of this knowledge from the review texts using \textit{natural language processing} (NLP) methods. In the papers by Cocarascu et al., the ontologies for films and hotels are mostly given, although mining some aspects from the review texts was also experimented with. The opinions of reviewers were extracted from the review texts using \textit{sentiment analysis} (SA) methods.
The objective of this project is to extend upon the work of Cocarascu et al. in order to design and implement a more generalised ADA to provide explanations for any product reviews on Amazon. As Amazon contains reviews for products ranging from pet supplies to digital music and car parts, this involves taking a more unsupervised and domain-independent approach to the various NLP tasks within the agent, as well as considering the effects of such variance on the dialogue between the user and the agent. Furthermore, ADA's complex methodology has never before been implemented within a fully developed end-to-end system, presenting a considerable software engineering challenge.
The objective of this project is to extend upon the work of Cocarascu et al.\ in order to design and implement a generalised ADA to provide explanations for any product reviews on Amazon. As Amazon contains reviews for products ranging from pet supplies to digital music and car parts, this involves taking a more unsupervised and domain-independent approach to the various NLP tasks within the agent, as well as considering the effects of such variance on the dialogue between the user and the agent. Furthermore, ADA's complex methodology has never before been implemented within a fully developed end-to-end system, presenting a considerable software engineering challenge.
\section{Contributions}
The main contributions of this project are threefold:
\begin{enumerate}
\item We propose and implement a novel method for domain-independent product ontology extraction, which allows us to build a \textit{feature-based representation} of any product in an unsupervised manner, an example of which is shown in Figure \ref{fig:toothbrushrepresentation} for wrist watches. The ontologies extracted with this method display significantly higher recall when compared to the widely used manually and autonomously annotated semantic networks WordNet and ConceptNet, respectively.
\item We implement a modified version of the method by Gao et al.\ \cite{RefWorks:doc:5ed3c3bbe4b0445db7f0a369} for feature-dependent sentiment analysis, which allows us to assess the reviewers' sentiment towards the various features of the product, whilst improving upon the accuracy of the phrase-level sentiment analysis in \cite{RefWorks:doc:5e08939de4b0912a82c3d46c}.
\item We implement ADA for the first time and from scratch as a fully developed system, with a backend agent including the aforementioned enhancements and a user-facing iOS chat application, which is shown in Figure \ref{fig:toothbrushscreenshot}.
\item We propose and implement a novel method for domain-independent product ontology extraction, which allows us to build a \textit{feature-based representation} of any product in an unsupervised manner, an example of which is shown in Figure \ref{fig:toothbrushrepresentation} for wrist watches. The ontologies extracted with this method obtain significantly higher F-1 scores when compared to the widely used manually and autonomously annotated semantic networks WordNet and ConceptNet, respectively.
\item We implement a modified version of the method by Gao et al.\ \cite{RefWorks:doc:5ed3c3bbe4b0445db7f0a369} for feature-level sentiment analysis, which allows us to assess the reviewers' sentiment towards the various features of the product, whilst improving upon the accuracy of the phrase-level sentiment analysis in \cite{RefWorks:doc:5e08939de4b0912a82c3d46c}.
\item We implement ADA for the first time and from scratch as a fully developed system, with a backend agent including the aforementioned enhancements and a user-facing iOS chat application, which is exemplified for a wrist watch in Figure \ref{fig:toothbrushscreenshot}.
\end{enumerate}
For both 1.\ and 2.\ we use BERT, a language model proposed by Devlin et al.\ \cite{RefWorks:doc:5e8dcdc3e4b0dba02bfdfa80}, which has produced state-of-the-art results in a wide variety of different NLP tasks.
For both 1.\ and 2.\ we use BERT, a language model proposed by Devlin et al.\ \cite{RefWorks:doc:5e8dcdc3e4b0dba02bfdfa80}, which has produced state-of-the-art results in a wide variety of different NLP tasks. We call the resulting system ADA-X for its ability to generalise for any product X.
\vspace{2cm}
......@@ -50,15 +50,15 @@ For both 1.\ and 2.\ we use BERT, a language model proposed by Devlin et al.\ \c
for tree = {draw, l=2.5cm}
[watch
[band
[links]
[leather]
[clasp]
[links, l=2cm]
[leather, l=2cm]
[clasp, l=2cm]
]
[face
[size]
[color]
[numbers]
[hands]
[size, l=2cm]
[color, l=2cm]
[numbers, l=2cm]
[hands, l=2cm]
]
[price]
[quality]
......@@ -68,14 +68,14 @@ For both 1.\ and 2.\ we use BERT, a language model proposed by Devlin et al.\ \c
]
\end{forest}
\caption{Feature-based representation for a wrist watch}
\caption{Ontology for a wrist watch}
\label{fig:toothbrushrepresentation}
\end{figure}
\begin{figure}[b]
\centering
\frame{\includegraphics[height=15cm]{images/watch_screenshot.png}}
\caption{A conversation between the user and ADA on the iOS app about a wrist watch}
\caption{A conversation between the user and ADA-X on the iOS app about a wrist watch}
\label{fig:toothbrushscreenshot}
\end{figure}
......
......@@ -57,7 +57,7 @@
\theoremstyle{def}
\newtheorem{definition}{Definition}[chapter]
\title{ADA Loves BERT: A System for Domain-Independent Feature-Based Product Review Aggregation}
\title{ADA-X: A System for Domain-Independent Feature-Based Product Review Aggregation}
\author{Joel Oksanen}
% Update supervisor and other title stuff in title/title.tex
......@@ -78,10 +78,6 @@ First of all, I would like to express my deepest gratitude to Professor Francesc
\noindent
I would also like to thank Ran Ji, Noora Kovalainen, Jenny Pan, and Nina Zhao for helping me evaluate the various aspects of ADA-X.
\vspace{\baselineskip}
\noindent
Finally, I would like to thank my parents, who afforded me this degree, and my friends, who helped me get through it.
\end{abstract}
\tableofcontents
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment