Muhammad Abdul-Mageed

Deep learning based models often achieve state-of-the-art performance in a wide range of natural language processing (NLP) tasks, which include open-ended tasks (e.g., story generation, brainstorming, and chat) and closed-ended tasks (e.g., summarization, question answering, and rewriting). To further enhance quality, there is a growing interest in scaling the model size and the amount of data used for training. These research efforts often overlook the impact of footprint metrics, such as high latency, high memory usage, and high energy consumption, on these deep learning models. A high footprint makes these models significantly inefficient for deployment on servers and devices such as tablets, handhelds, and wearables. Methods for improving model efficiency often come at the cost of degrading model quality.In this dissertation, we address the central question: how can we push the envelope in improving the efficiency-quality tradeoff of deep learning models for on-device NLP tasks? To this end, we propose methods that take on-device efficiency constraints (e.g., ≤ 16 MB memory or ≤ 200 ms latency) to inform the design of the model architecture. We propose methods for the manual design of architecture for the auto-completion task (generate continuations for user-written prompts) that enjoy a better memory-accuracy tradeoff than existing auto-completion models (Chapter 2). Additionally, we introduce methods that can directly take efficiency constraints to automatically search for efficient sparsely activated architectures for machine translation tasks (Chapter 3) and efficient pretrained (task-agnostic) language modeling architectures (Chapter 4). Finally, in Chapter 5, we explore a novel use case of employing large language models to speed up architecture search, while maintaining the efficiency and quality of state-of-the-art neural architecture search algorithms.

View record

Natural Language Processing (NLP) emerges as a critical solution for analyzing, manipulating, and understanding human language automatically and computationally, enabling the processing of vast amounts of language data swiftly. Computational NLP systems utilize numerical matrices or vectors as inputs, necessitating the conversion of discrete language symbols into a continuous representation space. The efficacy of these continuous representations is pivotal for developing successful NLP systems. With the advent of attention mechanisms, attention-based models have been adopted to learn contextual language representations by pre-training with language modeling (LM) objectives on extensive textual corpora. Despite the proven effectiveness of attention-based pre-trained language models (PLMs) in learning sequence-level representations for various NLP tasks, the integration of social aspects into representation learning remains unexplored. Recent efforts have applied PLMs to derive user-level representations, aiming to enhance content-based recommendation systems' transferability and precision. However, challenges persist in encoding lengthy user engagement histories, capturing users' diverse interests, and generating precomputable user-level representations. This dissertation focuses on advancing language representation learning for sequence-level sociopragmatic meaning (SM) comprehension and user-level content-based recommendation. For sequence-level SM, we introduce a novel weakly supervised method for pretraining and fine-tuning language models (Chapter 2). To enhance representation quality further, we propose a new contrastive learning framework for pretraining LMs (Chapter 3). Our approach is extended to the multilingual domain, presenting a unified, massively multilingual evaluation benchmark for SM (Chapter 4), alongside a comprehensive evaluation of state-of-the-art large language models for SM understanding. Addressing the challenges in learning user-level representations for recommendation systems, Chapter 5 introduces a novel framework that incorporates multiple poly-attention layers and sparse attention mechanisms. This framework hierarchically fuses token-level embeddings of session-based user history texts using PLM, tackling the intricacies of recommendation systems.

View record

This dissertation centers on Natural Language Processing (NLP) for African languages, endeavoring to unravel the progress, challenges, and future prospects within this linguistic context. The research encompasses language identification and Natural Language Understanding (NLU), Natural Language Generation (NLG), and culminates in a comprehensive case study on machine translation.The first chapter introduces the problem statement, articulates the motivation for addressing theissue, and presents the innovative solutions developed throughout this research. Chapter two discusses intricate details of African languages, offering insights into the genealogical classification, linguistic landscape, and the challenges of multilingual NLP. Building upon this foundation, the third chapter advocates for an Afrocentric approach to technology development, emphasizing the significance of aligning technology with the cultural values and linguistic diversity of African communities. It addresses challenges such as data scarcity and representation bias, spotlighting community-driven initiatives aimed at advancing NLP in the region.The fourth chapter unveils AfroLID, a neural language identification tool designed for 517 Africanlanguages and language varieties, establishing itself as the new state-of-the-art solution for Africanlanguage identification.Chapter five introduces SERENGETI, a massively multilingual language model tailored to support517 African languages and language varieties. Evaluation on AfroNLU, an extensive benchmark forAfrican NLP, showcases SERENGETI’s superior performance, thereby paving the way for transformative research and development across a diverse linguistic landscape.The sixth chapter addresses NLG challenges in African languages, presenting Cheetah, a language model designed for 517 African languages. Comprehensive evaluations underscore Cheetah’s capacity to generate contextually relevant text across various African languages.The seventh chapter presents a case study on machine translation, focusing on Bare Nouns (BNs)translation from Yorùbá to English. This study illuminates the challenges posed by informationasymmetry in machine translation and provides insights into the linguistic capabilities of StatisticalMachine Translation (SMT) and Neural Machine Translation (NMT) systems. Emphasizing theimportance of fine-grained linguistic considerations, the study encourages further research in addressing translation challenges faced by languages with BNs, analytic languages, and low-resource languages.In chapter eight, I conclude and discuss possible directions for future work.

View record