-
Universal Facial Encoding of Codec Avatars from VR Headsets
Authors:
Shaojie Bai,
Te-Li Wang,
Chenghui Li,
Akshay Venkatesh,
Tomas Simon,
Chen Cao,
Gabriel Schwartz,
Ryan Wrench,
Jason Saragih,
Yaser Sheikh,
Shih-En Wei
Abstract:
Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of he…
▽ More
Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our approach to prior face-encoding methods demonstrating significant improvements in both quantitative metrics and qualitative results.
△ Less
Submitted 17 July, 2024;
originally announced July 2024.
-
The Emergence of Large Language Models in Static Analysis: A First Look through Micro-Benchmarks
Authors:
Ashwin Prasad Shivarpatna Venkatesh,
Samkutty Sabu,
Amir M. Mir,
Sofia Reis,
Eric Bodden
Abstract:
The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT seri…
▽ More
The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT series and open-source models such as LLaMA. Our study reveals that LLMs show promising results in type inference, demonstrating higher accuracy than traditional methods, yet they exhibit limitations in callgraph analysis. This contrast emphasizes the need for specialized fine-tuning of LLMs to better suit specific static analysis tasks. Our findings provide a foundation for further research towards integrating LLMs for static analysis tasks.
△ Less
Submitted 27 February, 2024;
originally announced February 2024.
-
Hidden Gems in the Rough: Computational Notebooks as an Uncharted Oasis for IDEs
Authors:
Sergey Titov,
Konstantin Grotov,
Ashwin Prasad S. Venkatesh
Abstract:
In this paper, we outline potential ways for the further development of computational notebooks in Integrated Development Environments (IDEs). We discuss notebooks integration with IDEs, focusing on three main areas: facilitating experimentation, adding collaborative features, and improving code comprehension. We propose that better support of notebooks will not only benefit the notebooks, but als…
▽ More
In this paper, we outline potential ways for the further development of computational notebooks in Integrated Development Environments (IDEs). We discuss notebooks integration with IDEs, focusing on three main areas: facilitating experimentation, adding collaborative features, and improving code comprehension. We propose that better support of notebooks will not only benefit the notebooks, but also enhance IDEs by supporting new development processes native to notebooks. In conclusion, we suggest that adapting IDEs for more experimentation-oriented notebook processes will prepare them for the future of AI-powered programming.
△ Less
Submitted 21 February, 2024;
originally announced February 2024.
-
TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference Tools
Authors:
Ashwin Prasad Shivarpatna Venkatesh,
Samkutty Sabu,
Jiawei Wang,
Amir M. Mir,
Li Li,
Eric Bodden
Abstract:
In light of the growing interest in type inference research for Python, both researchers and practitioners require a standardized process to assess the performance of various type inference techniques. This paper introduces TypeEvalPy, a comprehensive micro-benchmarking framework for evaluating type inference tools. TypeEvalPy contains 154 code snippets with 845 type annotations across 18 categori…
▽ More
In light of the growing interest in type inference research for Python, both researchers and practitioners require a standardized process to assess the performance of various type inference techniques. This paper introduces TypeEvalPy, a comprehensive micro-benchmarking framework for evaluating type inference tools. TypeEvalPy contains 154 code snippets with 845 type annotations across 18 categories that target various Python features. The framework manages the execution of containerized tools, transforms inferred types into a standardized format, and produces meaningful metrics for assessment. Through our analysis, we compare the performance of six type inference tools, highlighting their strengths and limitations. Our findings provide a foundation for further research and optimization in the domain of Python type inference.
△ Less
Submitted 2 January, 2024; v1 submitted 28 December, 2023;
originally announced December 2023.
-
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations
Authors:
Karthik Gopalakrishnan,
Behnam Hedayatnia,
Qinlang Chen,
Anna Gottardi,
Sanjeev Kwatra,
Anu Venkatesh,
Raefer Gabriel,
Dilek Hakkani-Tur
Abstract:
Building socialbots that can have deep, engaging open-domain conversations with humans is one of the grand challenges of artificial intelligence (AI). To this end, bots need to be able to leverage world knowledge spanning several domains effectively when conversing with humans who have their own world knowledge. Existing knowledge-grounded conversation datasets are primarily stylized with explicit…
▽ More
Building socialbots that can have deep, engaging open-domain conversations with humans is one of the grand challenges of artificial intelligence (AI). To this end, bots need to be able to leverage world knowledge spanning several domains effectively when conversing with humans who have their own world knowledge. Existing knowledge-grounded conversation datasets are primarily stylized with explicit roles for conversation partners. These datasets also do not explore depth or breadth of topical coverage with transitions in conversations. We introduce Topical-Chat, a knowledge-grounded human-human conversation dataset where the underlying knowledge spans 8 broad topics and conversation partners don't have explicitly defined roles, to help further research in open-domain conversational AI. We also train several state-of-the-art encoder-decoder conversational models on Topical-Chat and perform automated and human evaluation for benchmarking.
△ Less
Submitted 23 August, 2023;
originally announced August 2023.
-
Trusting Language Models in Education
Authors:
Jogi Suda Neto,
Li Deng,
Thejaswi Raya,
Reza Shahbazi,
Nick Liu,
Adhitya Venkatesh,
Miral Shah,
Neeru Khosla,
Rodrigo Capobianco Guido
Abstract:
Language Models are being widely used in Education. Even though modern deep learning models achieve very good performance on question-answering tasks, sometimes they make errors. To avoid misleading students by showing wrong answers, it is important to calibrate the confidence - that is, the prediction probability - of these models. In our work, we propose to use an XGBoost on top of BERT to outpu…
▽ More
Language Models are being widely used in Education. Even though modern deep learning models achieve very good performance on question-answering tasks, sometimes they make errors. To avoid misleading students by showing wrong answers, it is important to calibrate the confidence - that is, the prediction probability - of these models. In our work, we propose to use an XGBoost on top of BERT to output the corrected probabilities, using features based on the attention mechanism. Our hypothesis is that the level of uncertainty contained in the flow of attention is related to the quality of the model's response itself.
△ Less
Submitted 7 August, 2023;
originally announced August 2023.
-
Learning to Detect Slip through Tactile Estimation of the Contact Force Field and its Entropy
Authors:
Xiaohai Hu,
Aparajit Venkatesh,
Yusen Wan,
Guiliang Zheng,
Neel Jawale,
Navneet Kaur,
Xu Chen,
Paul Birkmeyer
Abstract:
Detection of slip during object grasping and manipulation plays a vital role in object handling. Existing solutions primarily rely on visual information to devise a strategy for grasping. However, for robotic systems to attain a level of proficiency comparable to humans, especially in consistently handling and manipulating unfamiliar objects, integrating artificial tactile sensing is increasingly…
▽ More
Detection of slip during object grasping and manipulation plays a vital role in object handling. Existing solutions primarily rely on visual information to devise a strategy for grasping. However, for robotic systems to attain a level of proficiency comparable to humans, especially in consistently handling and manipulating unfamiliar objects, integrating artificial tactile sensing is increasingly essential. We introduce a novel physics-informed, data-driven approach to detect slip continuously in real time. We employ the GelSight Mini, an optical tactile sensor, attached to custom-designed grippers to gather tactile data. Our work leverages the inhomogeneity of tactile sensor readings during slip events to develop distinctive features and formulates slip detection as a classification problem. To evaluate our approach, we test multiple data-driven models on 10 common objects under different loading conditions, textures, and materials. Our results show that the best classification algorithm achieves a high average accuracy of 95.61%. We further illustrate the practical application of our research in dynamic robotic manipulation tasks, where our real-time slip detection and prevention algorithm is implemented.
△ Less
Submitted 28 April, 2024; v1 submitted 1 March, 2023;
originally announced March 2023.
-
Static Analysis Driven Enhancements for Comprehension in Machine Learning Notebooks
Authors:
Ashwin Prasad Shivarpatna Venkatesh,
Samkutty Sabu,
Mouli Chekkapalli,
Jiawei Wang,
Li Li,
Eric Bodden
Abstract:
Jupyter notebooks enable developers to interleave code snippets with rich-text and in-line visualizations. Data scientists use Jupyter notebook as the de-facto standard for creating and sharing machine-learning based solutions, primarily written in Python. Recent studies have demonstrated, however, that a large portion of Jupyter notebooks available on public platforms are undocumented and lacks a…
▽ More
Jupyter notebooks enable developers to interleave code snippets with rich-text and in-line visualizations. Data scientists use Jupyter notebook as the de-facto standard for creating and sharing machine-learning based solutions, primarily written in Python. Recent studies have demonstrated, however, that a large portion of Jupyter notebooks available on public platforms are undocumented and lacks a narrative structure. This reduces the readability of these notebooks. To address this shortcoming, this paper presents HeaderGen, a novel tool-based approach that automatically annotates code cells with categorical markdown headers based on a taxonomy of ML operations, and classifies and displays function calls according to this taxonomy. For this functionality to be realized, HeaderGen enhances an existing call graph analysis in PyCG. To improve precision, HeaderGen extends PyCG's analysis with support for handling external library code and flow-sensitivity. The former is realized by facilitating the resolution of function return-types. The evaluation on 15 real-world Jupyter notebooks from Kaggle shows that HeaderGen's underlying call graph analysis yields high accuracy (95.6% precision and 95.3% recall). This is because HeaderGen can resolve return-types of external libraries where existing type inference tools such as pytype (by Google), pyright (by Microsoft), and Jedi fall short. The header generation has a precision of 85.7% and a recall rate of 92.8%. In a user study, HeaderGen helps participants finish comprehension and navigation tasks faster. To further evaluate the type inference capability of tools, we introduce TypeEvalPy, a framework for evaluating type inference tools with a micro-benchmark containing 154 code snippets and 845 type annotations. Our comparative analysis on four tools revealed that HeaderGen outperforms other tools in exact matches with the ground truth.
△ Less
Submitted 11 June, 2024; v1 submitted 11 January, 2023;
originally announced January 2023.
-
Cross-domain Variational Capsules for Information Extraction
Authors:
Akash Nagaraj,
Akhil K,
Akshay Venkatesh,
Srikanth HR
Abstract:
In this paper, we present a characteristic extraction algorithm and the Multi-domain Image Characteristics Dataset of characteristic-tagged images to simulate the way a human brain classifies cross-domain information and generates insight. The intent was to identify prominent characteristics in data and use this identification mechanism to auto-generate insight from data in other unseen domains. A…
▽ More
In this paper, we present a characteristic extraction algorithm and the Multi-domain Image Characteristics Dataset of characteristic-tagged images to simulate the way a human brain classifies cross-domain information and generates insight. The intent was to identify prominent characteristics in data and use this identification mechanism to auto-generate insight from data in other unseen domains. An information extraction algorithm is proposed which is a combination of Variational Autoencoders (VAEs) and Capsule Networks. Capsule Networks are used to decompose images into their individual features and VAEs are used to explore variations on these decomposed features. Thus, making the model robust in recognizing characteristics from variations of the data. A noteworthy point is that the algorithm uses efficient hierarchical decoding of data which helps in richer output interpretation. Noticing a dearth in the number of datasets that contain visible characteristics in images belonging to various domains, the Multi-domain Image Characteristics Dataset was created and made publicly available. It consists of thousands of images across three domains. This dataset was created with the intent of introducing a new benchmark for fine-grained characteristic recognition tasks in the future.
△ Less
Submitted 13 October, 2022;
originally announced October 2022.
-
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Authors:
Aarohi Srivastava,
Abhinav Rastogi,
Abhishek Rao,
Abu Awal Md Shoeb,
Abubakar Abid,
Adam Fisch,
Adam R. Brown,
Adam Santoro,
Aditya Gupta,
Adrià Garriga-Alonso,
Agnieszka Kluska,
Aitor Lewkowycz,
Akshat Agarwal,
Alethea Power,
Alex Ray,
Alex Warstadt,
Alexander W. Kocurek,
Ali Safaya,
Ali Tazarv,
Alice Xiang,
Alicia Parrish,
Allen Nie,
Aman Hussain,
Amanda Askell,
Amanda Dsouza
, et al. (426 additional authors not shown)
Abstract:
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-futur…
▽ More
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
△ Less
Submitted 12 June, 2023; v1 submitted 9 June, 2022;
originally announced June 2022.
-
Automatic detection of glaucoma via fundus imaging and artificial intelligence: A review
Authors:
Lauren Coan,
Bryan Williams,
Krishna Adithya Venkatesh,
Swati Upadhyaya,
Silvester Czanner,
Rengaraj Venkatesh,
Colin E. Willoughby,
Srinivasan Kavitha,
Gabriela Czanner
Abstract:
Glaucoma is a leading cause of irreversible vision impairment globally and cases are continuously rising worldwide. Early detection is crucial, allowing timely intervention which can prevent further visual field loss. To detect glaucoma, examination of the optic nerve head via fundus imaging can be performed, at the centre of which is the assessment of the optic cup and disc boundaries. Fundus ima…
▽ More
Glaucoma is a leading cause of irreversible vision impairment globally and cases are continuously rising worldwide. Early detection is crucial, allowing timely intervention which can prevent further visual field loss. To detect glaucoma, examination of the optic nerve head via fundus imaging can be performed, at the centre of which is the assessment of the optic cup and disc boundaries. Fundus imaging is non-invasive and low-cost; however, the image examination relies on subjective, time-consuming, and costly expert assessments. A timely question to ask is can artificial intelligence mimic glaucoma assessments made by experts. Namely, can artificial intelligence automatically find the boundaries of the optic cup and disc (providing a so-called segmented fundus image) and then use the segmented image to identify glaucoma with high accuracy. We conducted a comprehensive review on artificial intelligence-enabled glaucoma detection frameworks that produce and use segmented fundus images. We found 28 papers and identified two main approaches: 1) logical rule-based frameworks, based on a set of simplistic decision rules; and 2) machine learning/statistical modelling based frameworks. We summarise the state-of-art of the two approaches and highlight the key hurdles to overcome for artificial intelligence-enabled glaucoma detection frameworks to be translated into clinical practice.
△ Less
Submitted 12 April, 2022;
originally announced April 2022.
-
Brushless Motor Performance Optimization by Eagle Strategy with Firefly and PSO
Authors:
Appalabathula Venkatesh,
Pradeepa H,
Chidanandappa R,
Shankar Nalinakshan,
Jayasankar V N
Abstract:
Brushless motors has special place though different motors are available because of its special features like absence in commutation, reduced noise and longer lifetime etc., The experimental parameter tracking of BLDC Motor can be achieved by developing a Reference system and their stability is guaranteed by adopting Lyapunov Stability theorems. But the stability is guaranteed only if the adaptive…
▽ More
Brushless motors has special place though different motors are available because of its special features like absence in commutation, reduced noise and longer lifetime etc., The experimental parameter tracking of BLDC Motor can be achieved by developing a Reference system and their stability is guaranteed by adopting Lyapunov Stability theorems. But the stability is guaranteed only if the adaptive system is incorporated with the powerful and efficient optimization techniques. In this paper the powerful eagle strategy with Particle Swarm optimization and Firefly algorithms are applied to evaluate the performance of brushless motor Where, Eagle Strategy(ES) with the use of Levys walk distribution function performs diversified global search and the Particle Swarm Optimization (PSO) and Firefly Algorithm(FFA) performs the efficient intensive local search. The combined operation makes the overall optimization technique as much convenient The simulation results are obtained by using MATLAB Simulink software
△ Less
Submitted 17 June, 2021;
originally announced June 2021.
-
Schema-Guided Natural Language Generation
Authors:
Yuheng Du,
Shereen Oraby,
Vittorio Perera,
Minmin Shen,
Anjali Narayan-Chen,
Tagyoung Chung,
Anu Venkatesh,
Dilek Hakkani-Tur
Abstract:
Neural network based approaches to data-to-text natural language generation (NLG) have gained popularity in recent years, with the goal of generating a natural language prompt that accurately realizes an input meaning representation. To facilitate the training of neural network models, researchers created large datasets of paired utterances and their meaning representations. However, the creation…
▽ More
Neural network based approaches to data-to-text natural language generation (NLG) have gained popularity in recent years, with the goal of generating a natural language prompt that accurately realizes an input meaning representation. To facilitate the training of neural network models, researchers created large datasets of paired utterances and their meaning representations. However, the creation of such datasets is an arduous task and they mostly consist of simple meaning representations composed of slot and value tokens to be realized. These representations do not include any contextual information that an NLG system can use when trying to generalize, such as domain information and descriptions of slots and values. In this paper, we present the novel task of Schema-Guided Natural Language Generation (SG-NLG). Here, the goal is still to generate a natural language prompt, but in SG-NLG, the input MRs are paired with rich schemata providing contextual information. To generate a dataset for SG-NLG we re-purpose an existing dataset for another task: dialog state tracking, which includes a large and rich schema spanning multiple different attributes, including information about the domain, user intent, and slot descriptions. We train different state-of-the-art models for neural natural language generation on this dataset and show that in many cases, including rich schema information allows our models to produce higher quality outputs both in terms of semantics and diversity. We also conduct experiments comparing model performance on seen versus unseen domains, and present a human evaluation demonstrating high ratings for overall output quality.
△ Less
Submitted 4 November, 2020; v1 submitted 11 May, 2020;
originally announced May 2020.
-
Optimizing Nondecomposable Data Dependent Regularizers via Lagrangian Reparameterization offers Significant Performance and Efficiency Gains
Authors:
Sathya N. Ravi,
Abhay Venkatesh,
Glenn Moo Fung,
Vikas Singh
Abstract:
Data dependent regularization is known to benefit a wide variety of problems in machine learning. Often, these regularizers cannot be easily decomposed into a sum over a finite number of terms, e.g., a sum over individual example-wise terms. The $F_β$ measure, Area under the ROC curve (AUCROC) and Precision at a fixed recall (P@R) are some prominent examples that are used in many applications. We…
▽ More
Data dependent regularization is known to benefit a wide variety of problems in machine learning. Often, these regularizers cannot be easily decomposed into a sum over a finite number of terms, e.g., a sum over individual example-wise terms. The $F_β$ measure, Area under the ROC curve (AUCROC) and Precision at a fixed recall (P@R) are some prominent examples that are used in many applications. We find that for most medium to large sized datasets, scalability issues severely limit our ability in leveraging the benefits of such regularizers. Importantly, the key technical impediment despite some recent progress is that, such objectives remain difficult to optimize via backpropapagation procedures. While an efficient general-purpose strategy for this problem still remains elusive, in this paper, we show that for many data-dependent nondecomposable regularizers that are relevant in applications, sizable gains in efficiency are possible with minimal code-level changes; in other words, no specialized tools or numerical schemes are needed. Our procedure involves a reparameterization followed by a partial dualization -- this leads to a formulation that has provably cheap projection operators. We present a detailed analysis of runtime and convergence properties of our algorithm. On the experimental side, we show that a direct use of our scheme significantly improves the state of the art IOU measures reported for MSCOCO Stuff segmentation dataset.
△ Less
Submitted 26 September, 2019;
originally announced September 2019.
-
Generating Accurate Pseudo-labels in Semi-Supervised Learning and Avoiding Overconfident Predictions via Hermite Polynomial Activations
Authors:
Vishnu Suresh Lokhande,
Songwong Tasneeyapant,
Abhay Venkatesh,
Sathya N. Ravi,
Vikas Singh
Abstract:
Rectified Linear Units (ReLUs) are among the most widely used activation function in a broad variety of tasks in vision. Recent theoretical results suggest that despite their excellent practical performance, in various cases, a substitution with basis expansions (e.g., polynomials) can yield significant benefits from both the optimization and generalization perspective. Unfortunately, the existing…
▽ More
Rectified Linear Units (ReLUs) are among the most widely used activation function in a broad variety of tasks in vision. Recent theoretical results suggest that despite their excellent practical performance, in various cases, a substitution with basis expansions (e.g., polynomials) can yield significant benefits from both the optimization and generalization perspective. Unfortunately, the existing results remain limited to networks with a couple of layers, and the practical viability of these results is not yet known. Motivated by some of these results, we explore the use of Hermite polynomial expansions as a substitute for ReLUs in deep networks. While our experiments with supervised learning do not provide a clear verdict, we find that this strategy offers considerable benefits in semi-supervised learning (SSL) / transductive learning settings. We carefully develop this idea and show how the use of Hermite polynomials based activations can yield improvements in pseudo-label accuracies and sizable financial savings (due to concurrent runtime benefits). Further, we show via theoretical analysis, that the networks (with Hermite activations) offer robustness to noise and other attractive mathematical properties.
△ Less
Submitted 31 March, 2020; v1 submitted 12 September, 2019;
originally announced September 2019.
-
Security Implications Of Compiler Optimizations On Cryptography -- A Review
Authors:
A. P. Shivarpatna Venkatesh,
A. Bhat Handadi,
M. Mory
Abstract:
When implementing secure software, developers must ensure certain requirements, such as the erasure of secret data after its use and execution in real time. Such requirements are not explicitly captured by the C language and could potentially be violated by compiler optimizations. As a result, developers typically use indirect methods to hide their code's semantics from the compiler and avoid unwa…
▽ More
When implementing secure software, developers must ensure certain requirements, such as the erasure of secret data after its use and execution in real time. Such requirements are not explicitly captured by the C language and could potentially be violated by compiler optimizations. As a result, developers typically use indirect methods to hide their code's semantics from the compiler and avoid unwanted optimizations. However, such workarounds are not permanent solutions, as increasingly efficient compiler optimization causes code that was considered secure in the past now vulnerable. This paper is a literature review of (1) the security complications caused by compiler optimizations, (2) approaches used by developers to mitigate optimization problems, and (3) recent academic efforts towards enabling security engineers to communicate implicit security requirements to the compiler. In addition, we present a short study of six cryptographic libraries and how they approach the issue of ensuring security requirements. With this paper, we highlight the need for software developers and compiler designers to work together in order to design efficient systems for writing secure software.
△ Less
Submitted 4 July, 2019;
originally announced July 2019.
-
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
Authors:
Sanghyun Yi,
Rahul Goel,
Chandra Khatri,
Alessandra Cervone,
Tagyoung Chung,
Behnam Hedayatnia,
Anu Venkatesh,
Raefer Gabriel,
Dilek Hakkani-Tur
Abstract:
Encoder-decoder based neural architectures serve as the basis of state-of-the-art approaches in end-to-end open domain dialog systems. Since most of such systems are trained with a maximum likelihood~(MLE) objective they suffer from issues such as lack of generalizability and the generic response problem, i.e., a system response that can be an answer to a large number of user utterances, e.g., "Ma…
▽ More
Encoder-decoder based neural architectures serve as the basis of state-of-the-art approaches in end-to-end open domain dialog systems. Since most of such systems are trained with a maximum likelihood~(MLE) objective they suffer from issues such as lack of generalizability and the generic response problem, i.e., a system response that can be an answer to a large number of user utterances, e.g., "Maybe, I don't know." Having explicit feedback on the relevance and interestingness of a system response at each turn can be a useful signal for mitigating such issues and improving system quality by selecting responses from different approaches. Towards this goal, we present a system that evaluates chatbot responses at each dialog turn for coherence and engagement. Our system provides explicit turn-level dialog quality feedback, which we show to be highly correlated with human evaluation. To show that incorporating this feedback in the neural response generation models improves dialog quality, we present two different and complementary mechanisms to incorporate explicit feedback into a neural response generation model: reranking and direct modification of the loss function during training. Our studies show that a response generation model that incorporates these combined feedback mechanisms produce more engaging and coherent responses in an open-domain spoken dialog setting, significantly improving the response quality using both automatic and human evaluation.
△ Less
Submitted 21 November, 2019; v1 submitted 29 April, 2019;
originally announced April 2019.
-
Natural Language Generation at Scale: A Case Study for Open Domain Question Answering
Authors:
Alessandra Cervone,
Chandra Khatri,
Rahul Goel,
Behnam Hedayatnia,
Anu Venkatesh,
Dilek Hakkani-Tur,
Raefer Gabriel
Abstract:
Current approaches to Natural Language Generation (NLG) for dialog mainly focus on domain-specific, task-oriented applications (e.g. restaurant booking) using limited ontologies (up to 20 slot types), usually without considering the previous conversation context. Furthermore, these approaches require large amounts of data for each domain, and do not benefit from examples that may be available for…
▽ More
Current approaches to Natural Language Generation (NLG) for dialog mainly focus on domain-specific, task-oriented applications (e.g. restaurant booking) using limited ontologies (up to 20 slot types), usually without considering the previous conversation context. Furthermore, these approaches require large amounts of data for each domain, and do not benefit from examples that may be available for other domains. This work explores the feasibility of applying statistical NLG to scenarios requiring larger ontologies, such as multi-domain dialog applications or open-domain question answering (QA) based on knowledge graphs. We model NLG through an Encoder-Decoder framework using a large dataset of interactions between real-world users and a conversational agent for open-domain QA. First, we investigate the impact of increasing the number of slot types on the generation quality and experiment with different partitions of the QA data with progressively larger ontologies (up to 369 slot types). Second, we perform multi-task learning experiments between open-domain QA and task-oriented dialog, and benchmark our model on a popular NLG dataset. Moreover, we experiment with using the conversational context as an additional input to improve response generation quality. Our experiments show the feasibility of learning statistical NLG models for open-domain QA with larger ontologies.
△ Less
Submitted 23 September, 2019; v1 submitted 19 March, 2019;
originally announced March 2019.
-
Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize
Authors:
Chandra Khatri,
Behnam Hedayatnia,
Anu Venkatesh,
Jeff Nunn,
Yi Pan,
Qing Liu,
Han Song,
Anna Gottardi,
Sanjeev Kwatra,
Sanju Pancholi,
Ming Cheng,
Qinglang Chen,
Lauren Stubel,
Karthik Gopalakrishnan,
Kate Bland,
Raefer Gabriel,
Arindam Mandal,
Dilek Hakkani-Tur,
Gene Hwang,
Nate Michel,
Eric King,
Rohit Prasad
Abstract:
Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog mo…
▽ More
Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog models, leveraging knowledge graphs for language understanding, handling complex utterances, building statistical and hierarchical dialog managers, and leveraging model-driven signals from user responses. The 2018 competition also included the provision of a suite of tools and models to the competitors including the CoBot (conversational bot) toolkit, topic and dialog act detection models, conversation evaluators, and a sensitive content detection model so that the competing teams could focus on building knowledge-rich, coherent and engaging multi-turn dialog systems. This paper outlines the advances developed by the university teams as well as the Alexa Prize team to achieve the common goal of advancing the science of Conversational AI. We address several key open-ended problems such as conversational speech recognition, open domain natural language understanding, commonsense reasoning, statistical dialog management, and dialog evaluation. These collaborative efforts have driven improved experiences by Alexa users to an average rating of 3.61, the median duration of 2 mins 18 seconds, and average turns to 14.6, increases of 14%, 92%, 54% respectively since the launch of the 2018 competition. For conversational speech recognition, we have improved our relative Word Error Rate by 55% and our relative Entity Error Rate by 34% since the launch of the Alexa Prize. Socialbots improved in quality significantly more rapidly in 2018, in part due to the release of the CoBot toolkit.
△ Less
Submitted 27 December, 2018;
originally announced December 2018.
-
Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervision
Authors:
Chandra Khatri,
Behnam Hedayatnia,
Rahul Goel,
Anushree Venkatesh,
Raefer Gabriel,
Arindam Mandal
Abstract:
As open-ended human-chatbot interaction becomes commonplace, sensitive content detection gains importance. In this work, we propose a two stage semi-supervised approach to bootstrap large-scale data for automatic sensitive language detection from publicly available web resources. We explore various data selection methods including 1) using a blacklist to rank online discussion forums by the level…
▽ More
As open-ended human-chatbot interaction becomes commonplace, sensitive content detection gains importance. In this work, we propose a two stage semi-supervised approach to bootstrap large-scale data for automatic sensitive language detection from publicly available web resources. We explore various data selection methods including 1) using a blacklist to rank online discussion forums by the level of their sensitiveness followed by randomly sampling utterances and 2) training a weakly supervised model in conjunction with the blacklist for scoring sentences from online discussion forums to curate a dataset. Our data collection strategy is flexible and allows the models to detect implicit sensitive content for which manual annotations may be difficult. We train models using publicly available annotated datasets as well as using the proposed large-scale semi-supervised datasets. We evaluate the performance of all the models on Twitter and Toxic Wikipedia comments testsets as well as on a manually annotated spoken language dataset collected during a large scale chatbot competition. Results show that a model trained on this collected data outperforms the baseline models by a large margin on both in-domain and out-of-domain testsets, achieving an F1 score of 95.5% on an out-of-domain testset compared to a score of 75% for models trained on public datasets. We also showcase that large scale two stage semi-supervision generalizes well across multiple classes of sensitivities such as hate speech, racism, sexual and pornographic content, etc. without even providing explicit labels for these classes, leading to an average recall of 95.5% versus the models trained using annotated public datasets which achieve an average recall of 73.2% across seven sensitive classes on out-of-domain testsets.
△ Less
Submitted 30 November, 2018;
originally announced November 2018.
-
Contextual Topic Modeling For Dialog Systems
Authors:
Chandra Khatri,
Rahul Goel,
Behnam Hedayatnia,
Angeliki Metanillou,
Anushree Venkatesh,
Raefer Gabriel,
Arindam Mandal
Abstract:
Accurate prediction of conversation topics can be a valuable signal for creating coherent and engaging dialog systems. In this work, we focus on context-aware topic classification methods for identifying topics in free-form human-chatbot dialogs. We extend previous work on neural topic classification and unsupervised topic keyword detection by incorporating conversational context and dialog act fe…
▽ More
Accurate prediction of conversation topics can be a valuable signal for creating coherent and engaging dialog systems. In this work, we focus on context-aware topic classification methods for identifying topics in free-form human-chatbot dialogs. We extend previous work on neural topic classification and unsupervised topic keyword detection by incorporating conversational context and dialog act features. On annotated data, we show that incorporating context and dialog acts leads to relative gains in topic classification accuracy by 35% and on unsupervised keyword detection recall by 11% for conversational interactions where topics frequently span multiple utterances. We show that topical metrics such as topical depth is highly correlated with dialog evaluation metrics such as coherence and engagement implying that conversational topic models can predict user satisfaction. Our work for detecting conversation topics and keywords can be used to guide chatbots towards coherent dialog.
△ Less
Submitted 18 October, 2018; v1 submitted 18 October, 2018;
originally announced October 2018.
-
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat
Authors:
Ravi Shekhar,
Aashish Venkatesh,
Tim Baumgärtner,
Elia Bruni,
Barbara Plank,
Raffaella Bernardi,
Raquel Fernández
Abstract:
We propose a grounded dialogue state encoder which addresses a foundational issue on how to integrate visual grounding with dialogue system components. As a test-bed, we focus on the GuessWhat?! game, a two-player game where the goal is to identify an object in a complex visual scene by asking a sequence of yes/no questions. Our visually-grounded encoder leverages synergies between guessing and as…
▽ More
We propose a grounded dialogue state encoder which addresses a foundational issue on how to integrate visual grounding with dialogue system components. As a test-bed, we focus on the GuessWhat?! game, a two-player game where the goal is to identify an object in a complex visual scene by asking a sequence of yes/no questions. Our visually-grounded encoder leverages synergies between guessing and asking questions, as it is trained jointly using multi-task learning. We further enrich our model via a cooperative learning regime. We show that the introduction of both the joint architecture and cooperative learning lead to accuracy improvements over the baseline system. We compare our approach to an alternative system which extends the baseline with reinforcement learning. Our in-depth analysis shows that the linguistic skills of the two models differ dramatically, despite approaching comparable performance levels. This points at the importance of analyzing the linguistic output of competing systems beyond numeric comparison solely based on task success.
△ Less
Submitted 15 March, 2019; v1 submitted 10 September, 2018;
originally announced September 2018.
-
Contextual Language Model Adaptation for Conversational Agents
Authors:
Anirudh Raju,
Behnam Hedayatnia,
Linda Liu,
Ankur Gandhe,
Chandra Khatri,
Angeliki Metallinou,
Anu Venkatesh,
Ariya Rastrow
Abstract:
Statistical language models (LM) play a key role in Automatic Speech Recognition (ASR) systems used by conversational agents. These ASR systems should provide a high accuracy under a variety of speaking styles, domains, vocabulary and argots. In this paper, we present a DNN-based method to adapt the LM to each user-agent interaction based on generalized contextual information, by predicting an opt…
▽ More
Statistical language models (LM) play a key role in Automatic Speech Recognition (ASR) systems used by conversational agents. These ASR systems should provide a high accuracy under a variety of speaking styles, domains, vocabulary and argots. In this paper, we present a DNN-based method to adapt the LM to each user-agent interaction based on generalized contextual information, by predicting an optimal, context-dependent set of LM interpolation weights. We show that this framework for contextual adaptation provides accuracy improvements under different possible mixture LM partitions that are relevant for both (1) Goal-oriented conversational agents where it's natural to partition the data by the requested application and for (2) Non-goal oriented conversational agents where the data can be partitioned using topic labels that come from predictions of a topic classifier. We obtain a relative WER improvement of 3% with a 1-pass decoding strategy and 6% in a 2-pass decoding framework, over an unadapted model. We also show up to a 15% relative improvement in recognizing named entities which is of significant value for conversational ASR systems.
△ Less
Submitted 31 July, 2018; v1 submitted 26 June, 2018;
originally announced June 2018.
-
Ask No More: Deciding when to guess in referential visual dialogue
Authors:
Ravi Shekhar,
Tim Baumgartner,
Aashish Venkatesh,
Elia Bruni,
Raffaella Bernardi,
Raquel Fernandez
Abstract:
Our goal is to explore how the abilities brought in by a dialogue manager can be included in end-to-end visually grounded conversational agents. We make initial steps towards this general goal by augmenting a task-oriented visual dialogue model with a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to…
▽ More
Our goal is to explore how the abilities brought in by a dialogue manager can be included in end-to-end visually grounded conversational agents. We make initial steps towards this general goal by augmenting a task-oriented visual dialogue model with a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to make a guess. Our analyses show that adding a decision making component produces dialogues that are less repetitive and that include fewer unnecessary questions, thus potentially leading to more efficient and less unnatural interactions.
△ Less
Submitted 12 June, 2018; v1 submitted 17 May, 2018;
originally announced May 2018.
-
On Evaluating and Comparing Open Domain Dialog Systems
Authors:
Anu Venkatesh,
Chandra Khatri,
Ashwin Ram,
Fenfei Guo,
Raefer Gabriel,
Ashish Nagar,
Rohit Prasad,
Ming Cheng,
Behnam Hedayatnia,
Angeliki Metallinou,
Rahul Goel,
Shaohua Yang,
Anirudh Raju
Abstract:
Conversational agents are exploding in popularity. However, much work remains in the area of non goal-oriented conversations, despite significant growth in research interest over recent years. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar university competition where sixteen selected university teams built conversational agents to deliv…
▽ More
Conversational agents are exploding in popularity. However, much work remains in the area of non goal-oriented conversations, despite significant growth in research interest over recent years. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar university competition where sixteen selected university teams built conversational agents to deliver the best social conversational experience. Alexa Prize provided the academic community with the unique opportunity to perform research with a live system used by millions of users. The subjectivity associated with evaluating conversations is key element underlying the challenge of building non-goal oriented dialogue systems. In this paper, we propose a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human judgement. The proposed metrics provide granular analysis of the conversational agents, which is not captured in human ratings. We show that these metrics can be used as a reasonable proxy for human judgment. We provide a mechanism to unify the metrics for selecting the top performing agents, which has also been applied throughout the Alexa Prize competition. To our knowledge, to date it is the largest setting for evaluating agents with millions of conversations and hundreds of thousands of ratings from users. We believe that this work is a step towards an automatic evaluation process for conversational AIs.
△ Less
Submitted 26 December, 2018; v1 submitted 10 January, 2018;
originally announced January 2018.
-
Topic-based Evaluation for Conversational Bots
Authors:
Fenfei Guo,
Angeliki Metallinou,
Chandra Khatri,
Anirudh Raju,
Anu Venkatesh,
Ashwin Ram
Abstract:
Dialog evaluation is a challenging problem, especially for non task-oriented dialogs where conversational success is not well-defined. We propose to evaluate dialog quality using topic-based metrics that describe the ability of a conversational bot to sustain coherent and engaging conversations on a topic, and the diversity of topics that a bot can handle. To detect conversation topics per utteran…
▽ More
Dialog evaluation is a challenging problem, especially for non task-oriented dialogs where conversational success is not well-defined. We propose to evaluate dialog quality using topic-based metrics that describe the ability of a conversational bot to sustain coherent and engaging conversations on a topic, and the diversity of topics that a bot can handle. To detect conversation topics per utterance, we adopt Deep Average Networks (DAN) and train a topic classifier on a variety of question and query data categorized into multiple topics. We propose a novel extension to DAN by adding a topic-word attention table that allows the system to jointly capture topic keywords in an utterance and perform topic classification. We compare our proposed topic based metrics with the ratings provided by users and show that our metrics both correlate with and complement human judgment. Our analysis is performed on tens of thousands of real human-bot dialogs from the Alexa Prize competition and highlights user expectations for conversational bots.
△ Less
Submitted 10 January, 2018;
originally announced January 2018.
-
Conversational AI: The Science Behind the Alexa Prize
Authors:
Ashwin Ram,
Rohit Prasad,
Chandra Khatri,
Anu Venkatesh,
Raefer Gabriel,
Qing Liu,
Jeff Nunn,
Behnam Hedayatnia,
Ming Cheng,
Ashish Nagar,
Eric King,
Kate Bland,
Amanda Wartick,
Yi Pan,
Han Song,
Sk Jayadevan,
Gene Hwang,
Art Pettigrue
Abstract:
Conversational agents are exploding in popularity. However, much work remains in the area of social conversation as well as free-form conversation over a broad range of domains and topics. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar university competition where sixteen selected university teams were challenged to build conversational…
▽ More
Conversational agents are exploding in popularity. However, much work remains in the area of social conversation as well as free-form conversation over a broad range of domains and topics. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar university competition where sixteen selected university teams were challenged to build conversational agents, known as socialbots, to converse coherently and engagingly with humans on popular topics such as Sports, Politics, Entertainment, Fashion and Technology for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research with a live system used by millions of users. The competition provided university teams with real user conversational data at scale, along with the user-provided ratings and feedback augmented with annotations by the Alexa team. This enabled teams to effectively iterate and make improvements throughout the competition while being evaluated in real-time through live user interactions. To build their socialbots, university teams combined state-of-the-art techniques with novel strategies in the areas of Natural Language Understanding, Context Modeling, Dialog Management, Response Generation, and Knowledge Acquisition. To support the efforts of participating teams, the Alexa Prize team made significant scientific and engineering investments to build and improve Conversational Speech Recognition, Topic Tracking, Dialog Evaluation, Voice User Experience, and tools for traffic management and scalability. This paper outlines the advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI.
△ Less
Submitted 10 January, 2018;
originally announced January 2018.
-
Optimal Tuning of Two-Dimensional Keyboards
Authors:
Aricca Bannerman,
James Emington,
Anil Venkatesh
Abstract:
We give a new analysis of a tuning problem in music theory, pertaining specifically to the approximation of harmonics on a two-dimensional keyboard. We formulate the question as a linear programming problem on families of constraints and provide exact solutions for many new keyboard dimensions. We also show that an optimal tuning for harmonic approximation can be obtained for any keyboard of given…
▽ More
We give a new analysis of a tuning problem in music theory, pertaining specifically to the approximation of harmonics on a two-dimensional keyboard. We formulate the question as a linear programming problem on families of constraints and provide exact solutions for many new keyboard dimensions. We also show that an optimal tuning for harmonic approximation can be obtained for any keyboard of given width, provided sufficiently many rows of octaves.
△ Less
Submitted 14 November, 2017;
originally announced November 2017.
