Deep Learning for Semantic and Syntactic Structures

Abstract

Deep machine learning has enjoyed recent success in vision and speech-to-text tasks, using deep multi-layered neural networks. They have obtained remarkable results particularly where the internal representation of the task is unclear. In parsing, where the structure of syntax is well studied and understood from linguistics, neural networks have so far not performed so well. State-of-the-art parsers use a tree-based graphical model that requires a large number of equivalent classes to represent each parse node and its phrase label. A recursive neural network (RNN) parser has been developed that works well on short sentences, but falls short of the state-of-the-art results on longer sentences. This thesis aims to investigate deep learning and improve parsing by examining how neural networks could perform state-of-the-art parsing by comparison with PCFG parsers. We hypothesize that a neural network could be configured to implement an algorithm parallel to PCFG parsers, and examine their suitability to this task from an analytic perspective. This highlights a missing term that the RNN parser is unable to model, and we identify the role of this missing term in parsing. We finally present two methods to improve the RNN parser by building upon the analysis in earlier chapters, one using an iterative process similar to belief propagation that yields a 0.38% improvement and another replacing the scoring method with a deeper neural model yielding a 0.83% improvement. By examining an RNN parser as an exemplar of a deep neural network, we gain insights to deep machine learning and some of the approximations it must make by comparing it with well studied non-neural parsers that achieve state-of-the-art results. In this way, our research provides a better understanding of deep machine learning and a step towards improvements in parsing that will lead to smarter algorithms that can learn more accurate representations of information and the syntax and semantics of text.