A Computational Framework to Improve Accessibility to Sanskrit Texts

Sanskrit is one of the most ancient and linguistically progressive languages that includes extensive written literature covering various fields such as religion, philosophy, sciences, and literature, whose digitalization has been increasingly growing in recent years. Although many digitized Sanskrit sources have become accessible, the use of these resources computationally faces many difficulties, including complex morphology, extensive compounding, abundant sandhi transformations, and lack of clear lexical boundaries. As sandhi transformations lead to variation of the constituent words by means of phonological changes, it becomes difficult to investigate the compound words unambiguously, resulting in an inability to use natural language processing tasks. In addition to that, traditional keyword-based retrieval systems cannot identify semantically connected texts written in other lexical, grammatical, or syntactic forms. These challenges highlight the need for computational methods that not only recover the internal structure of Sanskrit compound words but also enable meaningful semantic exploration of large digital Sanskrit corpora. This thesis presents a computational framework that enables accessibility to Sanskrit texts by solving two interconnected problems: automatic splitting and segmentation of Sanskrit compounds, as well as semantic information retrieval beyond exact-match searches. Instead of solving the two tasks separately, the research adopts a collective framework in which each step not only addresses an important limitation of existing methods but also serves as a basis for future improvements. Starting with boundary prediction on a character level, the framework gradually advances towards compound segmentation, uncertainty-aware modeling, and semantic information retrieval at a sentence level. The first contribution addresses the task of accurately locating the sandhi boundary in single-split Sanskrit compound words. A novel attention-based bi-encoder (ABBIE) architecture is introduced to predict semantically meaningful splits by leveraging contextually informed representations of characters. Building on this contribution, the second proposes a joint learning approach to simultaneously address boundary prediction and compound segmentation within a unified framework. Learning both tasks within a unified architecture allows mitigating the error propagation inherent to traditional pipelined solutions and increases segmentation consistency. The third contribution tackles the more complex task of multi-split compound processing where the number of hidden sandhi boundaries is not known a priori. Multi-Split Prediction of Sanskrit Compounds by Attention-based Bi-Encoders (ABBIE-M), an iterative boundary prediction approach, is introduced to correctly locate multiple splits without a priori knowledge of the number of constituent words. In the fourth contribution, compound segmentation is improved through a novel uncertainty-aware joint modeling for Sanskrit compound splitting and segmentation that combines the contextually informed character representations obtained with the Transformer architecture with sequence modeling provided by the BiLSTM neural network and Conditional Random Fields (CRFs). Such an approach explicitly accounts for uncertainty of predictions during the boundary detection step, thus increasing the segmentation reliability of linguistically ambiguous compounds. The last contribution of the thesis lies outside the domain of compound analysis. A novel framework for dense semantic retrieval of Sanskrit sentences is presented where the texts are embedded into a common contextual representation space and efficient meaning-based retrieval is performed via dense vector indexing. The proposed techniques are evaluated on publicly available datasets, as well as newly created benchmark datasets. Experimental results in comparison with state-of-the-art methods indicate that the proposed architecture shows competitive or better results in different experimental settings. In addition, several ablation analyses, qualitative results, confusion matrices, visualization of the attention mechanism, threshold analysis, and difficulty-level-based experiments help to understand the efficacy of the proposed architectural components as well as validate the importance of contextual attention mechanism, joint learning, boundary prediction iteration, uncertainty modeling, and dense semantic representations in improving the performance of the system. Moreover, the semantic retrieval framework shows its practical feasibility through several qualitative examples of semantic retrieval of Sanskrit passages conceptually related to the query but without an exact match. Other than contributing to improving individual tasks of Natural Language Processing (NLP), the research presents a cohesive framework for the development of computational processing of Sanskrit texts, combining lexical and semantic processing of the text. The methodologies presented contribute to the field of Sanskrit NLP in the following way: they enhance automatic compound analysis, increase robustness against ambiguity in the process of segmentation, and allow meaning-based access to large collections of Sanskrit texts. In addition, the research is relevant to computational linguistics for morphologically complex and low-resource languages as well as the digital humanities. Overall, this thesis demonstrates that combining contextual neural representations, joint learning, structured sequence modeling, uncertainty-aware prediction, and dense semantic retrieval provides an effective and coherent approach to improving computational access to Sanskrit texts. The experimental findings show competitive performance in the evaluated settings while establishing a strong foundation for future research toward comprehensive Sanskrit language understanding and large-scale semantic exploration of the world's classical textual heritage.