Commit ca6bd0ee authored by Joel Oksanen's avatar Joel Oksanen
Browse files
parents ba2d99f3 12d26ff1
......@@ -12,20 +12,21 @@ class ConnectionManager: ObservableObject {
@Published var product = Product()
@Published var messages: [Message] = [Message]()
private let ip = "192.168.0.13"
private let ip = "192.168.1.104"
private let port = "8000"
private var messageQueue: [Message]? = nil
init() {
requestProduct(id: "B0075SUK14")
requestProduct(id: "B000AYW0M2", type: "watch")
// B00RTGK0N0 - red Canon camera
// B004J3V90Y - Canon T3i
// B0012YA85A - Canon Rebel XSI
// B003ZYF3LO - Nikon D3100
// B0075SUK14 - Backpack
}
private func requestProduct(id: String) {
let url = URL(string: "http://" + ip + ":" + port + "/ios_server/product/?id=" + id)!
private func requestProduct(id: String, type: String) {
let url = URL(string: "http://" + ip + ":" + port + "/ios_server/product/?id=" + id + "&type=" + type)!
let task = URLSession.shared.dataTask(with: url) {(data, response, error) in
guard let data = data else { return }
......
\chapter{Feature-dependent sentiment analysis}
\section{Exploration}
\subsection{Sentence-level SA trained on domain-specific data}
\subsection{Feature-dependent SA trained on SemEval-2014 data}
\section{Evaluation}
\ No newline at end of file
\appendix
\chapter{First Appendix}
\ No newline at end of file
\chapter{Extracted ontologies}
\label{sec:ontology_appendix}
\ No newline at end of file
......@@ -16,12 +16,12 @@ ADA is designed around a \textit{feature}-based conceptualisation of products. R
Figure \ref{fig:ADApipeline} presents an overview of how ADA works in the movie review domain. It takes as input reviews snippets for a particular movie, and outputs a dialectical strength measure for the movie along with dialogical explanations for the proposed strength. The ADA pipeline can be divided into three main functions:
\begin{enumerate}
\item ADA uses Natural Language Processing (NLP) methods to mine a a \textit{review aggregation} (RA) for the movie, which consists of a set of \textit{votes} $\mathcal{V}$ expressing the reviewers' sentiments on the movie and its features.
\item ADA uses Natural Language Processing (NLP) methods to mine a \textit{review aggregation} (RA) for the movie, which consists of a set of \textit{votes} $\mathcal{V}$ expressing the reviewers' sentiments on the movie and its features.
\item The review aggregation is used to generate a Quantitative Bipolar Argumentation Framework (QBAF), which summarises the reviewers' general consensus on each of the features.
\item \textit{Gradual semantics} are applied to the QBAF to calculate the strength measure and provide dialogical explanations.
\end{enumerate}
We will from now on refer to the object under review as a \textit{product} $p$, considering a more generic domain. Furthermore, we refer to $\mathcal{A} = \{p\} \cup \mathcal{F}$ as the set of $arguments$ upon which votes can act, where $\mathcal{F}$ is the product's feature set.
Considering a more generic domain, we will from now on refer to the object under review as a \textit{product} $p$. Furthermore, as reviewers can have opinions on the product as a whole or the product's \textit{features} $\mathcal{F}$, we refer to $\mathcal{A} = \{p\} \cup \mathcal{F}$ as the set of \textit{arguments} upon which votes can act.
In the following sections, we will go through the different aspects of ADA in more detail. We will conclude section 2.1 by discussing the limitations of ADA with specific regards to Amazon reviews.
......@@ -33,7 +33,7 @@ ADA is designed around a \textit{feature-based characterisation} of products:
Let $\mathcal{P}$ be a given set of products, and $p \in \mathcal{P}$ be any product. A feature-based characterisation of $p$ is a set $\mathcal{F}$ of features with sub-features $\mathcal{F}' \subset \mathcal{F}$ such that each $f' \in \mathcal{F}'$ has a unique parent $p(f') \in \mathcal{F}$; for any $f \in \mathcal{F} \backslash \mathcal{F}'$, we define $p(f) = p$.
\end{definition}
This feature-based characterisation can be \textit{predetermined}, i.e. obtained from metadata. For example, for a digital camera $p$, we could use metadata to obtain $\mathcal{F} = \{f_I,f_L,f_B\}$, where $f_I$ is \textit{image quality}, $f_L$ is \textit{lens}, and $f_B$ is \textit{battery}, as these are features that are present in most, if not all, digital cameras. Features can also be \textit{mined} from the reviews themselves using more sophisticated methods, such as the semantic network ConceptNet\footnote{http://conceptnet.io/} to identify related terms to either the product or its known features, in order to obtain further sub-features. For our camera example, we could mine the sub-features $f_{L1}'$ for \textit{zoom} and $f_{L2}'$ for \textit{autofocus}. Figure \ref{fig:featurebasedrepresentation} shows the full feature-based representation for our example.
This feature-based characterisation can be \textit{predetermined}, i.e. obtained from metadata. For example, for a digital camera $p$, we could use metadata to obtain $\mathcal{F} = \{f_I,f_L,f_B\}$, where $f_I$ is \textit{image quality}, $f_L$ is \textit{lens}, and $f_B$ is \textit{battery}, as these are features that are present in most, if not all, digital cameras. Features can also be \textit{mined} from the reviews themselves using more sophisticated methods, such as the semantic network ConceptNet\footnote{http://conceptnet.io/} to identify related terms to either the product or its known features, in order to obtain further sub-features. For our camera example, we could mine the sub-features $f_{L1}'$ for \textit{zoom} and $f_{L2}'$ for \textit{autofocus}. Figure \ref{fig:featurebasedrepresentation} shows the full feature-based representation for our running example.
\begin{figure}[h]
\centering
......@@ -206,7 +206,7 @@ Figure \ref{fig:QBAF} shows an example of a QBAF for a digital camera, with the
\path[->] (fl1) edge node {$-$} (fl);
\path[->] (fl2) edge node [label=above right:$+$] {} (fl);
\end{tikzpicture}
\caption{QBAF for a digital camera annotated with base scores $\tau$ and DF-QuAD strengths $\sigma$.}
\caption{QBAF for a digital camera annotated with base scores $\tau$ and DF-QuAD strengths $\sigma$}
\label{fig:QBAF}
\end{figure}
......@@ -364,9 +364,7 @@ Because all Rotten Tomatoes reviews are on movies, there are features such as \t
\paragraph{Writing style}
\hangindent=\parindent
\hangafter=0
As the reviews on Rotten Tomatoes are written by critics, the style of writing tends to be similar across all reviews. On the other hand, Amazon reviews can be written by anyone who has purchased the product, so the style is more individualistic. For example in the review for the \textit{Canon IXUS 185} digital camera, the user comments on their personal interests. Colloquial language (\textit{'specs'}) and spelling mistakes (\textit{'ot'}) are also common. Due to these differences, using the same off-the-shelf movie review classifier for sentiment analysis on Amazon reviews would not be sensible. \newline
%Sentiment Analysis choice: off-the-shelf classifier: too specific for this project's domain: does not distinguish between different polarities
As the reviews on Rotten Tomatoes are written by critics, the style of writing tends to be similar across all reviews. On the other hand, Amazon reviews can be written by anyone who has purchased the product, so the style is more individualistic. For example in the review for the \textit{Canon IXUS 185} digital camera, the user comments on their personal interests. Colloquial language (\textit{'specs'}) and spelling mistakes (\textit{'ot'}) are also common. Due to these differences, using the same off-the-shelf movie review classifier for sentiment analysis on Amazon reviews would not be sensible. Although ADA was explored in the user-written review domain of Trip Advisor in \cite{RefWorks:doc:5e0de20ee4b055d63d355913}, this paper focused on expanding the conversational aspect of ADA with no consideration of its NLP methods. \newline
Based on these two differences, we propose two extensions to ADA in order to accommodate Amazon reviews:
\begin{enumerate}
......@@ -376,10 +374,6 @@ Based on these two differences, we propose two extensions to ADA in order to acc
\section{Feature-level sentiment analysis}
\begin{center}
\fbox{\parbox{13cm}{\textbf{Note:} This section will be extended with details about the particular NLP methods I will use once I've went through neural networks in my machine learning course.}}
\end{center}
\noindent
In this section, we will examine state of the art research in \textit{aspect-level sentiment analysis} \cite{RefWorks:doc:5e2b0d8de4b0711bafe4fba8}, which attempts to determine people's opinions on \textit{entities} and their \textit{aspects}. We have already been introduced to aspect-level sentiment analysis in ADA's review aggregation, where the entities are \textit{products} and the aspects are their \textit{features}. For consistency, we will refer to entities as products and aspects as features. Further research into this area will guide our implementation of the extensions proposed in section 2.1.
......@@ -394,24 +388,24 @@ Since ADA would associate \textit{Sonicare toothbrush} with \textit{terrible}, i
\begin{enumerate}
\item cannot distinguish that the toothbrush under review is referred to as \textit{this model};
\item will calculate only a single sentiment for the entire review, while it actually represents a negative sentiment towards other Sonicare toothbrushes and a positive sentiment towards this particular toothbrush.
\item will calculate only a single sentiment for the entire phrase, while it actually represents a negative sentiment towards other Sonicare toothbrushes and a positive sentiment towards this particular toothbrush.
\end{enumerate}
The first error has to do with \textit{feature extraction}, which accounts for the first part of feature-level sentiment analysis where the agent extracts features from the text. The second error has to do with \textit{feature-dependent sentiment analysis}, where the extracted features are assigned with sentiment polarities. The following sections evaluate research in these two areas in hopes of finding a way to resolve these errors.
\subsection{Feature extraction}
In order to obtain an unsupervised feature-based representation for a product, we must have a way to extract the features from some source of information about the product, such as metadata or its review texts. As the availability of metadata varies greatly from one product category to another, we shall focus our analysis on feature extraction from the review texts in order to not limit the set of Amazon products supported by our implementation.
In order to obtain an unsupervised feature-based representation for a product, we must have a way to extract the features and their relations to one another from some source of information about the product, such as metadata or its review texts. As the availability of metadata on Amazon varies greatly from one product category to another, we shall focus our analysis on feature extraction from the review texts in order to not limit the set of Amazon products supported by our implementation.
\subsubsection{Opinion target extraction}
A lot of research has already gone into feature extraction from text using NLP methods in the form of \textit{opinion target extraction}. Most studies on this topic can be categorised into \textit{rule-based methods} or \textit{supervised machine learning based methods} \cite{RefWorks:doc:5e374c55e4b0d3ee568e80d6}. The former relies on lexical data or syntactic rules to detect features, such as in \cite{RefWorks:doc:5e38230ae4b07b376b61b3fe}, while the latter models the problem as a sequence labelling task with the use of Conditional Random Fields (CRFs) \cite{RefWorks:doc:5e381a1ce4b084bfe828c41a}. More recently, deep learning methods have also been proposed for the problem, an overview of which can be found in \cite{RefWorks:doc:5e2b0d8de4b0711bafe4fba8}.
In order to construct the feature-based representation of a product, we wish to extract the features of the product towards which the reviewers have an opinion. In the field of \textit{opinion target extraction}, various NLP methods are used to extract such words from natural language. Most studies on this topic can be categorised into \textit{rule-based methods} or \textit{supervised machine learning based methods} \cite{RefWorks:doc:5e374c55e4b0d3ee568e80d6}. The former relies on lexical data or syntactic rules to detect features, such as in \cite{RefWorks:doc:5e38230ae4b07b376b61b3fe}, while the latter models the problem as a sequence labelling task, often with the use of Conditional Random Fields (CRFs) \cite{RefWorks:doc:5e381a1ce4b084bfe828c41a}. More recently, deep learning methods have also been proposed for the problem, an overview of which can be found in \cite{RefWorks:doc:5e2b0d8de4b0711bafe4fba8}.
When it comes to Amazon review texts, the issue with many of these methods is that they are domain-dependent. Particularly for the machine learning methods, the models work well on the domain on which they are trained on, but may face a performance drop of up to $40\%$ when tested in different domains \cite{RefWorks:doc:5e374c55e4b0d3ee568e80d6}. Therefore, a model trained on digital camera reviews might not perform well on reviews for garden hoses. On the other hand, a model trained on the whole Amazon review dataset might be too general for individual domains.
When it comes to Amazon review texts, the issue with many of these methods is that they are domain-dependent. Particularly for the machine learning methods, the models work well on the domain on which they are trained on, but may face a performance drop of up to $40\%$ when tested in different domains \cite{RefWorks:doc:5e374c55e4b0d3ee568e80d6}. Therefore, a model trained on digital camera reviews might not perform well on reviews for garden hoses. On the other hand, it would be impossible to obtain manually labelled data that would be representative of the whole selection of millions of Amazon products.
Some cross-domain feature extraction methods have been proposed \cite{RefWorks:doc:5e374c55e4b0d3ee568e80d6}, but these still perform significantly worse than the single-domain methods discussed above. Furthermore, none of these methods take advantage of the semantic relationships between the product and its features, but look for opinion targets in the absence of any existing product information.
\subsubsection{ConceptNet}
\subsubsection{ConceptNet and WordNet}
Semantic information can be obtained from \textit{ConceptNet} \cite{RefWorks:doc:5e382bf3e4b0034ec2324aed}, which is a graph connecting words and phrases with labelled and weighted edges expressing semantic relations between the words. As our goal is to obtain features of a product, the relations we are most interested in are:
\begin{itemize}
......@@ -420,28 +414,79 @@ Semantic information can be obtained from \textit{ConceptNet} \cite{RefWorks:doc
\item \textit{MadeOf} for materials of products;
\item \textit{UsedFor} for uses of products.
\end{itemize}
For example, the term \textit{electric toothbrush}\footnote{http://conceptnet.io/c/en/electric\_toothbrush} is related by the \textit{UsedFor} relation to \textit{cleanliness} and by the \textit{CapableOf} relation to \textit{run on batteries}. However, many reviews for electronic toothbrushes comment on the product \textit{model} or its timer functionality, neither of which are terms related to electronic toothbrushes on ConceptNet. Due to the incompleteness of ConceptNet, a hybrid approach to feature extraction where opinion targets are mined from review texts for wider categories of products to complement the product-specific semantic information from ConceptNet might be the most effective.
% Another possible source of information is the semantic network ConceptNet, which can identify related terms based on semantic information.
%\cite{RefWorks:doc:5e374c55e4b0d3ee568e80d6}
%Predetermined (metadata) \newline
%Mined (semantic network: ConceptNet: Feature categorization) \newline
%Mined (deep learning) \newline
%Representation tiers
For example, the term \textit{electric toothbrush}\footnote{http://conceptnet.io/c/en/electric\_toothbrush} is related by the \textit{UsedFor} relation to \textit{cleanliness} and by the \textit{CapableOf} relation to \textit{run on batteries}. However, many reviews for electronic toothbrushes comment on the product \textit{model} or its \textit{timer} functionality, neither of which are terms related to electronic toothbrushes on ConceptNet.
\textit{WordNet} \cite{RefWorks:doc:5ed503e0e4b081759f6d02e2}, a large manually constructed lexical database, is another popular source for semantic information between words. Although WordNet includes fewer relations than ConceptNet, it includes the relation of \textit{meronymy}, which denotes a word being a constituent part of another word, similar to the \textit{HasA} and \textit{MadeOf} relations in ConceptNet. For example, the term \textit{electric toothbrush}\footnote{http://wordnetweb.princeton.edu} has a single meronym of \textit{electric motor}. As with ConceptNet, the list of relations on WordNet is incomplete, as it fails to include relevant meronyms such as the timer.
Due to the low recall of ConceptNet and WordNet in obtaining product features, our approach to target extraction must take advantage of the review texts as a source of information about the product. However, we can evaluate our implementation against ConceptNet and WordNet, or possibly take a hybrid approach to feature extraction where features are mined from review texts to complement the existing semantic information on ConceptNet and WordNet.
\subsection{Feature-dependent sentiment analysis}
After we have extracted opinion targets (arguments) from a review, we wish to discern whether the opinions towards the arguments are positive or negative through sentiment analysis. Perhaps the main difficulty in feature-dependent sentiment analysis is to distinguish which opinions are acting on which arguments.
After we have extracted the features of a product, we wish to discern whether the opinions towards the features in reviews are positive or negative through sentiment analysis. Perhaps the main difficulty in feature-dependent sentiment analysis is to distinguish which opinions are acting on which arguments.
ADA attempts to tackle this issue by diving the review into phrases at specific keywords, such as the word \textit{but} in \textit{I liked the acting, but the cinematography was dreadful}, after which it assumes each phrase contains at most one sentiment. However, there are many cases where such a simple method will not work, like the example at the start of this section. This is particularly true for Amazon reviews where the language tends to be less formal compared to Rotten Tomatoes reviews written by professional critics.
More advanced methods using deep learning have been proposed in literature, although the task is deemed difficult and there is currently no dominating technique for this purpose \cite{RefWorks:doc:5e2b0d8de4b0711bafe4fba8}. Dong et al.\ \cite{RefWorks:doc:5e2e107ce4b0bc4691206e2e} proposed an \textit{adaptive recursive neural network} (AdaRNN) for target-dependent Twitter sentiment classification, which propagates the sentiments of words to the target by exploiting the context and the syntactic relationships between them. The results were compared with a re-implementation of \textit{SVM-dep} proposed by Jiang et al.\ \cite{RefWorks:doc:5e2e1e23e4b0e67b35d1c360}, which relies on target-dependent syntactic features in a SVM classifier instead of a neural network. The result for both methods are promising, and the domain of Twitter is similar to Amazon reviews in terms of formality.
However, both methods were trained and tested in the same domain including tweets about celebrities, companies and consumer electronics. The performance would likely drop substantially in a separate domain, as the sentiment polarity of a word can be highly dependent on context: for example the adjective \textit{hard} has a positive connotation when describing a protective case, but a negative connotation when describing an armchair.
\section{BERT}
Both of the two NLP tasks relevant to ADA, feature extraction and feature-dependent sentiment analysis, become more difficult to perform in a general domain, which is crucial to applying ADA for all Amazon products. BERT \cite{RefWorks:doc:5e8dcdc3e4b0dba02bfdfa80}, which stands for \textit{Bidirectional Encoder Representations from Transformers}, is a state-of-the-art language representation model, which uses pre-training to learn language representations from large amounts of unlabelled text, which can then be fine-tuned for more specific NLP tasks. Because the unlabelled text used to pre-train BERT comes from a general domain corpus such as Wikipedia, the knowledge in a pre-trained BERT model is especially well-suited for domain-independent NLP tasks such as ours. This section will provide a brief overview of the BERT architecture and the underlying \textit{Transformer network}, shown in Figure \ref{fig:BERTarchitecture}. The paper presents two model sizes for BERT, BERT-base and BERT-large, which differ in terms of the size of various hyperparameters. In this paper, we will use the former due to memory constraints.
\vspace{0.5cm}
\begin{figure}[h]
\centering
\includegraphics[width=14cm]{images/bert_architecture.png}
\caption{Overview of the BERT architecture}
\label{fig:BERTarchitecture}
\end{figure}
\subsection{Tokenizer}
The first step of BERT is to tokenise the input text in order to prepare it for the Transformer network, where each token follows its own path through the network. The tokenisation step in Figure \ref{fig:BERTarchitecture} is a bit simplified, as BERT uses \textit{WordPiece} tokenisation, which divides words into a limited set of common sub-word units, improving the handling of rare words. The first token is always the \texttt{[CLS]} token, which is used in the output as representative of the whole sequence of tokens. The \texttt{[SEP]} token is used to mark the end of the first sentence in tasks with two sentences, such as \textit{Sentence Pair Classification}. Finally, the sequence of tokens is padded to a fixed length of up to 512 tokens, which enables BERT to handle batches of texts with varying sizes. The token ids are then passed onto the Embedding layer of the Transformer network.
\subsection{Transformer network}
The pre-trained network of BERT is based on an Embedding layer followed by a stack of Transformer encoders, proposed by Vaswani et.\ al.\ \cite{RefWorks:doc:5e8dc8a4e4b010098c29b2b0}.
\subsubsection{Embedding layer}
Each token id from the tokeniser is passed through an embedding layer, which outputs a vector $E_n \in \mathbb{R}^H$ of hidden size $H$ (768 for BERT-base). The embedding is a sum of the \textit{token embedding}, a \textit{position embedding} for the token's position in the sequence, and a \textit{segment embedding} which is used to distinguish between the first and second sentences in Sentence Pair Classification. The embeddings for each token are then passed onto the first encoder in the encoder network.
\subsubsection{Encoder network}
The Encoder network consists of a stack of 12 identical encoders for BERT-base. The input to the first encoder are the embeddings, and the output from the $(n-1)$th encoder is used as input to the $n$th encoder for $n \in [2,12]$. Figure \ref{fig:BERTencoder} shows a more detailed diagram of a single encoder in the encoder network. Each encoder consists of two parts, a \textit{self-attention} layer and a feed-forward neural network.
Although the representations for each token follow their own path through the network, the self-attention layer is where they interact with each other in order to build a context-aware representation for each token. Let $X_{n-1} \in \mathbb{R}^{T,H}$ be a matrix containing the inputs to the encoder, where $T$ is the number of tokens. Each self-attention layer learns three weight matrices, $W_Q$, $W_K$, and $W_V$ used to calculate the $Q$ (query), $K$ (key), and $V$ (value) matrices respectively:
\begin{align*}
Q &= X_{n-1} \times W_Q \\
K &= X_{n-1} \times W_K \\
V &= X_{n-1} \times W_V.
\end{align*}
Then, the outputs $Z$ of the self-attention layer are calculated with the formula
$$Z = \texttt{softmax}(\frac{Q \times K^T}{\sqrt{d_k}}) \times V,$$
where $d_k$ an arbitrary constant used to stabilise the gradient. The calculation $Q \times K^T$ returns for each token position (row of the matrix) the query vector of that position multiplied with the key vectors of all of the positions. The \texttt{softmax} operation then normalises the resulting rows so that each sums up to one and represents how much each token position is expressed at the given position. Finally, we multiply the \texttt{softmax} probabilities for each position with the corresponding value vectors, in order to obtain the output $Z$. In essence, each row of $Z$ is a weighted combination of all of the value vectors in $V$ where the weights are determined by $Q$ and $K$.
\begin{figure}[b]
\centering
\includegraphics[width=11cm]{images/encoder.png}
\caption{A single encoder}
\label{fig:BERTencoder}
\end{figure}
\newpage
BERT implements \textit{multi-headed} attention, which means that each self-attention layer has in fact 12 (16 for BERT-large) sets of query, key, and value matrices which are all randomly initialised and learn a different representation subspace. Therefore, we end up with 12 output matrices $Z_1,\dots,Z_{12}$ which are concatenated and multiplied with an additional weight matrix $W_O$ to obtain the final output matrix $Z$ which captures information from all of the attention heads. Multi-headed attention improves the performance of the self-attention layer by allowing the model to focus on different positions in the different representation subspaces.
ADA attempts to tackle this issue by diving the review into phrases at specific keywords, such as the word \textit{but} in \textit{I liked the acting, but the cinematography was dreadful}, after which it assumes each phrase contains at most one sentiment. However, there are many cases where such a simple method will not work, like the example at the start of this section. This is particularly true for Amazon reviews where the text tends to be less formal compared to Rotten Tomatoes reviews.
The outputs of the self-attention layer are used as inputs to the feed-forward neural network, which returns outputs $X_{n} \in \mathbb{R}^{T,H}$ used as inputs to the next encoder. The same network is independently applied to each position, so the interaction between positions happens in the self-attention layer only.
More advanced methods using deep learning have been proposed in literature, although the task is deemed difficult and there is currently no dominating technique for this purpose \cite{RefWorks:doc:5e2b0d8de4b0711bafe4fba8}. Dong et al.\ \cite{RefWorks:doc:5e2e107ce4b0bc4691206e2e} proposed an \textit{adaptive recursive neural network} (AdaRNN) for target-dependent Twitter sentiment classification, which propagates the sentiments of words to the target by exploiting the context and the syntactic relationships between them. The result were promising, and the domain of Twitter is similar to Amazon reviews in terms of formality. The results were compared with a re-implementation of \textit{SVM-dep} proposed by Jiang et al.\ \cite{RefWorks:doc:5e2e1e23e4b0e67b35d1c360}, which uses target-dependent syntactic features in a SVM classifier instead of a neural network. Since SVM-dep performed nearly as well as AdaRNN, either could be used to improve ADAs sentiment analysis accuracy.
\subsection{Fine-tuning BERT}
The outputs from the last encoder, which are denoted in Figure \ref{fig:BERTarchitecture} as $C$ for the \texttt{[CLS]} token position and $T_n$ for the $n$th text token, are used as inputs to a classifier network depending on the specific NLP task. Many sentence-level classification tasks use simply the output $C$, which is considered representative of the whole sequence of tokens, while the outputs $T_n$ are more representative of the $n$th token position.
%Sentiment analysis vs. argument modelling \newline
%\cite{RefWorks:doc:5e08939de4b0912a82c3d46c} counter example: “although the screen is very clear, the battery life is too short.” \newline
%Aspect-level sentiment classification: "three important tasks in aspect-level sentiment classification using neural networks" \cite{RefWorks:doc:5e2b0d8de4b0711bafe4fba8} \newline
%AdaRNN \cite{RefWorks:doc:5e2e107ce4b0bc4691206e2e} vs. SVM-dep \cite{RefWorks:doc:5e2e1e23e4b0e67b35d1c360}
While the weights for BERT can be loaded from pre-trained models, the classifier is trained from scratch for each task in what is called \textit{fine-tuning} the model. Often, the weights in BERT are also fine-tuned alongside training the classifier, although the changes are minimal and BERT mostly relies on its pre-trained knowledge, which it passes onto the classifier.
\section{Conversational systems}
......
@inproceedings{RefWorks:doc:5edca760e4b0ef3565a5f38d,
author={Tomas Mikolov and Ilya Sutskever and Kai Chen and Greg S. Corrado and Jeff Dean},
year={2013},
title={Distributed representations of words and phrases and their compositionality},
booktitle={Advances in neural information processing systems},
pages={3111-3119}
}
@inproceedings{RefWorks:doc:5edc9ecbe4b03b813c4d4381,
author={Jianmo Ni and Jiacheng Li and Julian McAuley},
year={2019},
title={Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects},
booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
pages={188-197}
}
@article{RefWorks:doc:5edbafdde4b0482c79eb8d95,
author={Tomas Mikolov and Kai Chen and Greg Corrado and Jeffrey Dean},
year={2013},
title={Efficient estimation of word representations in vector space},
journal={arXiv preprint arXiv:1301.3781}
}
@article{RefWorks:doc:5ed503e0e4b081759f6d02e2,
author={George A. Miller},
year={1995},
month={nov},
title={WordNet: A Lexical Database for English},
journal={Commun.ACM},
volume={38},
number={11},
pages={39–41},
isbn={0001-0782},
url={https://doi.org/10.1145/219717.219748},
doi={10.1145/219717.219748}
}
@inproceedings{RefWorks:doc:5ebbb4fbe4b081d7661a5c8a,
author={Zhen Tan and Xiang Zhao and Wei Wang and Weidong Xiao},
year={2019},
title={Jointly extracting multiple triplets with multilayer translation constraints},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={33},
pages={7080-7087}
}
@misc{RefWorks:doc:5eba827fe4b04ec536ffa3a3,
author = {Kui Xue and Yangming Zhou and Zhiyuan Ma and Tong Ruan and Huanhuan Zhang and Ping He},
year = {2019},
title = {Fine-tuning BERT for Joint Entity and Relation Extraction in Chinese Medical Text},
journal = {arXiv preprint arXiv:1908.07721}
}
@inproceedings{RefWorks:doc:5eba7afee4b0f1b04f6dc725,
author={Tianwen Jiang and Tong Zhao and Bing Qin and Ting Liu and Nitesh Chawla and Meng Jiang},
year={2019},
title={Multi-Input Multi-Output Sequence Labeling for Joint Extraction of Fact and Condition Tuples from Scientific Text},
booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
pages={302-312}
}
@misc{RefWorks:doc:5eba6f57e4b005e3ba46d108,
author = {Suncong Zheng and Feng Wang and Hongyun Bao and Yuexing Hao and Peng Zhou and Bo Xu},
year = {2017},
title = {Joint extraction of entities and relations based on a novel tagging scheme},
journal = {arXiv preprint arXiv:1706.05075}
}
@misc{RefWorks:doc:5eba65d4e4b0b30b1e768373,
author = {Ling Luo and Zhihao Yang and Mingyu Cao and Lei Wang and Yin Zhang and Hongfei Lin},
year = {2020},
title = {A neural network-based joint learning approach for biomedical entity and relation extraction from biomedical literature},
journal = {Journal of Biomedical Informatics},
volume = {103},
pages = {103384},
note = {ID: 272371},
abstract = {Recently joint modeling methods of entity and relation exhibit more promising results than traditional pipelined methods in general domain. However, they are inappropriate for the biomedical domain due to numerous overlapping relations in biomedical text. To alleviate the problem, we propose a neural network-based joint learning approach for biomedical entity and relation extraction. In this approach, a novel tagging scheme that takes into account overlapping relations is proposed. Then the Att-BiLSTM-CRF model is built to jointly extract the entities and their relations with our extraction rules. Moreover, the contextualized ELMo representations pre-trained on biomedical text are used to further improve the performance. Experimental results on biomedical corpora show that our method can significantly improve the performance of overlapping relation extraction and achieves the state-of-the-art performance. "},
isbn = {1532-0464},
doi={https://doi.org/10.1016/j.jbi.2020.103384}
}
@misc{RefWorks:doc:5eb97f10e4b084c78e199bc3,
author = {Antoine Bosselut and Hannah Rashkin and Maarten Sap and Chaitanya Malaviya and Asli Celikyilmaz and Yejin Choi},
year = {2019},
title = {Comet: Commonsense transformers for automatic knowledge graph construction},
journal = {arXiv preprint arXiv:1906.05317}
}
@misc{RefWorks:doc:5eb97a3ae4b04ec536ff3ba1,
author = {S. Huang and X. Liu and X. Peng and Z. Niu},
year = {2012},
title = {Fine-grained Product Features Extraction and Categorization in Reviews Opinion Mining},
journal = {2012 IEEE 12th International Conference on Data Mining Workshops},
pages = {680-686},
note = {ID: 1},
isbn = {2375-9259},
doi={10.1109/ICDMW.2012.53}
}
@misc{RefWorks:doc:5eb97a1be4b05621fa57d477,
author = {Giulio Petrucci and Marco Rospocher and Chiara Ghidini},
year = {2018},
title = {Expressive ontology learning as neural machine translation},
journal = {Journal of Web Semantics},
volume = {52-53},
pages = {66-82},
note = {ID: 272989},
abstract = {Automated ontology learning from unstructured textual sources has been proposed in literature as a way to support the difficult and time-consuming task of knowledge modeling for semantic applications. In this paper we propose a system, based on a neural network in the encoder–decoder configuration, to translate natural language definitions into Description Logics formulæ through syntactic transformation. The model has been evaluated to assess its capacity to generalize over different syntactic structures, tolerate unknown words, and improve its performance by enriching the training set with new annotated examples. The results obtained in our evaluation show how approaching the ontology learning problem as a neural machine translation task can be a valid way to tackle long term expressive ontology learning challenges such as language variability, domain independence, and high engineering costs. "},
isbn = {1570-8268},
doi={https://doi.org/10.1016/j.websem.2018.10.002}
}
@misc{RefWorks:doc:5eb97898e4b084c78e1997f2,
author = {Zhijun Yan and Meiming Xing and Dongsong Zhang and Baizhang Ma},
year = {2015},
title = {EXPRS: An extended pagerank method for product feature extraction from online consumer reviews},
journal = {Information & Management},
volume = {52},
number = {7},
pages = {850-858},
note = {ID: 271670},
abstract = {Online consumer product reviews are a main source for consumers to obtain product information and reduce product uncertainty before making a purchase decision. However, the great volume of product reviews makes it tedious and ineffective for consumers to peruse individual reviews one by one and search for comments on specific product features of their interest. This study proposes a novel method called EXPRS that integrates an extended PageRank algorithm, synonym expansion, and implicit feature inference to extract product features automatically. The empirical evaluation using consumer reviews on three different products shows that EXPRS is more effective than two baseline methods. "},
isbn = {0378-7206},
doi={https://doi.org/10.1016/j.im.2015.02.002}
}
@misc{RefWorks:doc:5eb80ce5e4b081d766196d67,
author = {Peng Qi and Yuhao Zhang and Yuhui Zhang and Jason Bolton and Christopher D. Manning},
year = {2020},
title = {Stanza: A Python Natural Language Processing Toolkit for Many Human Languages},
journal = {arXiv preprint arXiv:2003.07082}
}
@article{RefWorks:doc:5eaebe76e4b098fe9e0217c2,
author={Artuur Leeuwenberg and Mihaela Vela and Jon Dehdari and Josef van Genabith},
year={2016},
title={A minimally supervised approach for synonym extraction with word embeddings},
journal={The Prague Bulletin of Mathematical Linguistics},
volume={105},
number={1},
pages={111-142}
}
@article{RefWorks:doc:5ea44ea2e4b026ebb6d0c04f,
author={Sarah Wiegreffe and Yuval Pinter},
year={2019},
title={Attention is not not Explanation},
journal={arXiv preprint arXiv:1908.04626}
}
@inproceedings{RefWorks:doc:5e96ec1ee4b0b3ce6e5b8fa8,
author={Joachim Wagner and Piyush Arora and Santiago Cortes and Utsab Barman and Dasha Bogdanova and Jennifer Foster and Lamia Tounsi},
year={2014},
month={aug},
title={DCU: Aspect-based Polarity Classification for SemEval Task 4},
booktitle={Proceedings of the 8th International Workshop on Semantic Evaluation ({S}em{E}val 2014)},
publisher={Association for Computational Linguistics},
address={Dublin, Ireland},
pages={223-229,},
url={https://www.aclweb.org/anthology/S14-2036},
doi={10.3115/v1/S14-2036"}
}
@article{RefWorks:doc:5e96c080e4b03874ac494b89,
author={Sarthak Jain and Byron C. Wallace},
year={2019},
title={Attention is not explanation},
journal={arXiv preprint arXiv:1902.10186}
}
@article{RefWorks:doc:5e8dcdc3e4b0dba02bfdfa80,
author={Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},
year={2018},
title={Bert: Pre-training of deep bidirectional transformers for language understanding},
journal={arXiv preprint arXiv:1810.04805}
}
@inproceedings{RefWorks:doc:5e8dc8a4e4b010098c29b2b0,
author={Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Łukasz Kaiser and Illia Polosukhin},
year={2017},
title={Attention is all you need},
booktitle={Advances in neural information processing systems},
pages={5998-6008}
}
@article{RefWorks:doc:5ed3c3bbe4b0445db7f0a369,
author={Zhengjie Gao and Ao Feng and Xinyu Song and Xi Wu},
year={2019},
title={Target-Dependent Sentiment Classification With BERT},
journal={IEEE Access},
volume={7},
pages={154290-154299}
}
@inproceedings{RefWorks:doc:5e6d25bee4b0b5553c7b4dab,
author={Maria Pontiki and Dimitris Galanis and Haris Papageorgiou and Ion Androutsopoulos and Suresh Manandhar and Mohammad AL-Smadi and Mahmoud Al-Ayyoub and Yanyan Zhao and Bing Qin and Orph 'ee De Clercq and V. 'eronique Hoste and Marianna Apidianaki and Xavier Tannier and Natalia Loukachevitch and Evgeniy Kotelnikov and Nuria Bel and Jim 'enez-Zafra, Salud Mar ' ia and Eryi ugit, G ul csen},
year={2016},
month={jun},
title={SemEval-2016 Task 5: Aspect Based Sentiment Analysis},
booktitle={Proceedings of the 10th International Workshop on Semantic Evaluation ({S}em{E}val-2016)},
publisher={Association for Computational Linguistics},
address={San Diego, California},
pages={19-30,},
url={https://www.aclweb.org/anthology/S16-1002},
doi={10.18653/v1/S16-1002"}
}
@inproceedings{RefWorks:doc:5e6d25a1e4b0e722bb9d840d,
author={Maria Pontiki and Dimitris Galanis and Haris Papageorgiou and Suresh Manandhar and Ion Androutsopoulos},
year={2015},
month={jun},
title={SemEval-2015 Task 12: Aspect Based Sentiment Analysis},
booktitle={Proceedings of the 9th International Workshop on Semantic Evaluation ({S}em{E}val 2015)},
publisher={Association for Computational Linguistics},
address={Denver, Colorado},
pages={486-495,},
url={https://www.aclweb.org/anthology/S15-2082},
doi={10.18653/v1/S15-2082"}
}
@misc{RefWorks:doc:5e3b01f7e4b0b4e3b49da908,
author = {Francesco Colace and Massimo De Santo and Luca Greco and Vincenzo Moscato and Antonio Picariello},
year = {2015},
title = {A collaborative user-centered framework for recommending items in Online Social Networks},
journal = {Computers in Human Behavior},
volume = {51},
pages = {694-704},
note = {ID: 271802},
abstract = {Recommender Systems are more and more playing an important role in our life, representing useful tools helping users to find “what they need” from a very large number of candidates and supporting people in making decisions in various contexts: what items to buy, which movie to watch, or even who they can invite to their social network, etc. In this paper, we propose a novel collaborative user-centered recommendation approach in which several aspects related to users and available in Online Social Networks – i.e. preferences (usually in the shape of items’ metadata), opinions (textual comments to which it is possible to associate a sentiment), behavior (in the majority of cases logs of past items’ observations made by users), feedbacks (usually expressed in the form of ratings) – are considered and integrated together with items’ features and context information within a general framework that can support different applications using proper customizations (e.g., recommendation of news, photos, movies, travels, etc.). Experiments on system accuracy and user satisfaction in several domains shows how our approach provides very promising and interesting results. "},
isbn = {0747-5632},
url = {http://www.sciencedirect.com/science/article/pii/S0747563214007262},
doi={https://doi.org/10.1016/j.chb.2014.12.011}
}
@article{RefWorks:doc:5e3afc64e4b019af6e2d8963,
author={Camila Vaccari Sundermann and Marcos Aur 'elio Domingues and Roberta Akemi Sinoara and Ricardo Marcondes Marcacini and Solange Oliveira Rezende},
year={2019},
title={Using Opinion Mining in Context-Aware Recommender Systems: A Systematic Review},
journal={Information},
volume={10},
number={2},
pages={42},
url={https://doi.org/10.3390/info10020042},
doi={10.3390/info10020042}
}
@inproceedings{RefWorks:doc:5e3af8a4e4b04ec55f2d8a5d,
author={Min Hou and Le Wu and Enhong Chen and Zhi Li and Vincent W. Zheng and Qi Liu},
year={2019},
title={Explainable Fashion Recommendation: A Semantic Attribute Region Guided Approach},
booktitle={Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, {IJCAI} 2019, Macao, China, August 10-16, 2019},
pages={4681-4688},
note={DBLP:conf/ijcai/2019},
url={https://doi.org/10.24963/ijcai.2019/650},
doi={10.24963/ijcai.2019/650}
}
@article{RefWorks:doc:5e3af891e4b07263889000bb,
author={Guoshuai Zhao and Hao Fu and Ruihua Song and Tetsuya Sakai and Zhongxia Chen and Xing Xie and Xueming Qian},
year={2019},
title={Personalized Reason Generation for Explainable Song Recommendation},
journal={ACM} {TIST}},
volume={10},
number={4},
pages={41:1-41:21},
url={https://doi.org/10.1145/3337967},
doi={10.1145/3337967}
}
@article{RefWorks:doc:5e3aedcbe4b07263888fff16,
author={Sophie de Kok and Linda Punt and Rosita van den Puttelaar and Karoliina Ranta and Kim Schouten and Flavius Frasincar},
year={2018},
title={Review-aggregated aspect-based sentiment analysis with ontology features},
journal={Progress in {AI}},
volume={7},
number={4},
pages={295-306},
url={https://doi.org/10.1007/s13748-018-0163-7},
doi={10.1007/s13748-018-0163-7}
}
@article{RefWorks:doc:5e39660de4b07263888fb37d,
author={R. Chulaka Gunasekara and David Nahamoo and Lazaros C. Polymenakos and Ciaurri, David Echeverr ' ia and Jatin Ganhotra and Kshitij P. Fadnis},
year={2019},
title={Quantized Dialog - A general approach for conversational systems},
journal={Computer Speech {\&} Language},
volume={54},
pages={17-30},
url={https://doi.org/10.1016/j.csl.2018.06.003},
doi={10.1016/j.csl.2018.06.003}
}
@inproceedings{RefWorks:doc:5e395ff6e4b02ac586d9a2c8,
author={Lorenz Cuno Klopfenstein and Saverio Delpriori and Silvia Malatini and Alessandro Bogliolo},
year={2017},
title={The Rise of Bots: A Survey of Conversational Interfaces, Patterns, and Paradigms},
booktitle={Proceedings of the 2017 Conference on Designing Interactive Systems, {DIS} '17, Edinburgh, United Kingdom, June 10-14, 2017},
pages={555-565},
note={DBLP:conf/ACMdis/2017},
url={https://doi.org/10.1145/3064663.3064672},
doi={10.1145/3064663.3064672}
}
@inproceedings{RefWorks:doc:5e382bf3e4b0034ec2324aed,
author={Robyn Speer and Joshua Chin and Catherine Havasi},
year={2017},
......@@ -56,38 +329,32 @@
url={https://doi.org/10.1145/2507157.2507163},
doi={10.1145/2507157.2507163}
}
@article{RefWorks:doc:5e39660de4b07263888fb37d,
author={R. Chulaka Gunasekara and David Nahamoo and Lazaros C. Polymenakos and Ciaurri, David Echeverr ' ia and Jatin Ganhotra and Kshitij P. Fadnis},
year={2019},
title={Quantized Dialog - A general approach for conversational systems},
journal={Computer Speech {\&} Language},
volume={54},
pages={17-30},
url={https://doi.org/10.1016/j.csl.2018.06.003},
doi={10.1016/j.csl.2018.06.003}
}
@inproceedings{RefWorks:doc:5e395ff6e4b02ac586d9a2c8,
author={Lorenz Cuno Klopfenstein and Saverio Delpriori and Silvia Malatini and Alessandro Bogliolo},
@inproceedings{RefWorks:doc:5e31af55e4b017f1b5fb8684,
author={Filip Radlinski and Nick Craswell},
year={2017},
title={The Rise of Bots: A Survey of Conversational Interfaces, Patterns, and Paradigms},
booktitle={Proceedings of the 2017 Conference on Designing Interactive Systems, {DIS} '17, Edinburgh, United Kingdom, June 10-14, 2017},
pages={555-565},
note={DBLP:conf/ACMdis/2017},
url={https://doi.org/10.1145/3064663.3064672},
doi={10.1145/3064663.3064672}
title={A Theoretical Framework for Conversational Search},
booktitle={Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval},
series={CHIIR ’17},
publisher={Association for Computing Machinery},
address={New York, NY, USA},
location={Oslo, Norway},
pages={117–126},
isbn={9781-450346771},
url={https://doi.org/10.1145/3020165.3020183},
doi={10.1145/3020165.3020183}
}
@article{RefWorks:doc:5e2f3970e4b0241a7d69e2a4,
author={Fatih Gedikli and Dietmar Jannach and Mouzhi Ge},
year={2014},
title={How should I explain? A comparison of different explanation types for recommender systems},
journal={International Journal of Human-Computer Studies},
volume={72},
number={4},
pages={367-382},
note={ID: 272548},
isbn={1071-5819},
url={http://www.sciencedirect.com/science/article/pii/S1071581913002024},
@misc{RefWorks:doc:5e2f3970e4b0241a7d69e2a4,
author = {Fatih Gedikli and Dietmar Jannach and Mouzhi Ge},
year = {2014},
title = {How should I explain? A comparison of different explanation types for recommender systems},
journal = {International Journal of Human-Computer Studies},
volume = {72},
number = {4},
pages = {367-382},
note = {ID: 272548},
abstract = {Recommender systems help users locate possible items of interest more quickly by filtering and ranking them in a personalized way. Some of these systems provide the end user not only with such a personalized item list but also with an explanation which describes why a specific item is recommended and why the system supposes that the user will like it. Besides helping the user understand the output and rationale of the system, the provision of such explanations can also improve the general acceptance, perceived quality, or effectiveness of the system. In recent years, the question of how to automatically generate and present system-side explanations has attracted increased interest in research. Today some basic explanation facilities are already incorporated in e-commerce Web sites such as Amazon.com. In this work, we continue this line of recent research and address the question of how explanations can be communicated to the user in a more effective way. In particular, we present the results of a user study in which users of a recommender system were provided with different types of explanation. We experimented with 10 different explanation types and measured their effects in different dimensions. The explanation types used in the study include both known visualizations from the literature as well as two novel interfaces based on tag clouds. Our study reveals that the content-based tag cloud explanations are particularly helpful to increase the user-perceived level of transparency and to increase user satisfaction even though they demand higher cognitive effort from the user. Based on these insights and observations, we derive a set of possible guidelines for designing or selecting suitable explanations for recommender systems. "},
isbn = {1071-5819},
url = {http://www.sciencedirect.com/science/article/pii/S1071581913002024},
doi={https://doi.org/10.1016/j.ijhcs.2013.12.007}
}
@inproceedings{RefWorks:doc:5e2e1e23e4b0e67b35d1c360,
......@@ -113,20 +380,6 @@
url={https://www.aclweb.org/anthology/P14-2009},
doi={10.3115/v1/P14-2009"}
}
@inproceedings{RefWorks:doc:5e31af55e4b017f1b5fb8684,
author={Filip Radlinski and Nick Craswell},
year={2017},
title={A Theoretical Framework for Conversational Search},
booktitle={Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval},
series={CHIIR ’17},
publisher={Association for Computing Machinery},