Projects

Content:

At MaiNLP we aim to make NLP models more robust, so that they can deal better with underlying shifts in data due to language variation.

On-going research projects

The following lists selected on-going research projects at MaiNLP, including selected publications:

ERC Consolidator grant DIALECT: Natural Language Understanding for non-standard languages and dialects

Dialects are ubiquitous and for many speakers are part of everyday life. They carry important social and communicative functions. Yet, dialects and non-standard languages in general are a blind spot in research on Natural Language Understanding (NLU). Despite recent breakthroughs, NLU still fails to take linguistic diversity into account. This lack of modeling language variation results in biased language models with high error rates on dialect data. This failure excludes millions of speakers today and prevents the development of future technology that can adapt to such users.

To account for linguistic diversity, a paradigm shift is needed: Away from data-hungry algorithms with passive learning from large data and single ground truth labels, which are known to be biased. To go past current learning practices, the key is to tackle variation at both ends: in input data and label bias. With DIALECT, I propose such an integrated approach, to devise algorithms which aid transfer from rich variability in inputs, and interactive learning which integrates human uncertainty in labels. This will reduce the need for data and enable better adaptation and generalization.

Advances in salient areas of deep learning research now make it possible to tackle this challenge. DIALECT’s objectives are to devise a) new algorithms and insights to address extremely scarce data setups and biased labels; b) novel representations which integrate auxiliary sources of information such as complement text data with speech; and c) new datasets with conversational data in its most natural form.

By integrating dialectal variation into models able to learn from scarce data and biased labels, the foundations will be established for fairer and more accurate NLU to break down language and literary barriers.

For further details about the project, please visit the DIALECT webpage.

Selected publications:

KLIMA-MEMES: The Impact of Humorous Communication on Political Decision-Making in the Climate Change Context

Climate change is a pressing problem facing humanity and a major polarizing topic in public discourse. Discussions on climate change pervade political agendas worldwide. IPCC experts agree that currently implemented measures against climate change are inadequate in their efforts. Information akin to this often rapidly breaks into social media spheres (e.g., Instagram and TikTok) in increasingly visual and often humor-driven attention cycles. The KLIMA-MEMES project will analyze how such communication in the form of memes or other visual media with an intent to be humorous can affect political decision-making. This is a collaborative project with multiple partners at LMU, funded by the Bavarian Research Institute for Digital Transformation.

More information:

On-going collaboration projects

The following lists selected on-going research projects at other institutions, which MaiNLP is a partner in:

Travolta: Tracing Attitudes And Variation In Online Luxembourgish Text Archives

This project, led by Prof. Christoph Purschke from the University of Luxembourg combines Natural Language Processing with sociolinguistic analysis to trace the development of written Luxembourgish and public discourse online.

Completed research projects

DFF Sapere Aude Project MultiVaLUe: Multilingual Variety-aware Language Understanding Technology

Intelligent machines that understand natural language texts are the Holy Grail of Artificial Intelligence. If achieved, they can automatically extract useful information from big messy text collections. Many challenges must be overcome first. To alleviate the scarcity of resources and broaden the scope to Danish and other small languages, we will unify two strands of research, transfer learning and weak supervision, with the aim to design cross-domain and cross-lingual algorithms that extract information more robustly under minimal guidance. In this project we work on two concrete applications: cross-lingual syntactic parsing (and representation learning on the linguistic manifold) and cross-domain information extraction.

Selected publications:

DFF Project thematic AI, MultiSkill: Multilingual Information Extraction for Job Posting Analysis (2020-2024)

Job markets are about to undergo profound changes in the years to come. The skills required to perform most jobs will shift significantly. This is due to a series of interrelated developments in technology, migration and digitization. As skills change, we are facing increasing needs for quicker and better hiring to better match people to jobs. Big multilingual job vacancy data are emerging on a variety and multitude of platforms. Such big data can provide insights on labor market skill set demands. This project is centered around computational job market analysis, to reliably perform high-precision information extraction on targeted domain data.

Selected publications:

Applications for PhD, Postdoc and student assistant positions

Interested in PhD, Postdoc or student assistants jobs? →Open positions



BSc/MSc thesis projects

Interested in doing your MSc or BSc thesis with us? We offer several BSc and MSc thesis topics at MaiNLP.

Currently, the following research vectors characterize broad topics in which we offer MSc and BSc thesis projects. We provide a (non-exhaustive) list of research projects within each vector. We are also interested to supervise projects related to the Research projects indicated above. You are welcome to send us your own project proposal. We recommend to check out the suggested/selected publications to get inspired.

Unless otherwise specified, all projects can be either at the MSc or BSc level. The exact scope of the project will be determined in a meeting before the start of the project.

Important note: We currently do not supervise industrial MSc/BSc thesis project (Industrieabschlussarbeiten).

Regularly check back here for updates on thesis project suggestions.

News:

  • 2024, Sep 1: MSc/BSc project applications deadlines posted. See list of project below and how to apply here

Legend:

  • :hourglass_flowing_sand: topic currently reserved
  • strikethrough topic no longer available

MSc/BSc thesis research vectors:

V1: NLP for Dialects, Multilinguality

Selected research projects

  • NLP for dialectal/non-standard language data. You are welcome to propose thesis projects related to processing dialectal or non-standard language data. Selected example projects are given below. Please include the following information in your application: (1) the dialect(s)/language(s) you are interested in working with and your familiarity with them, (2) one or more NLP tasks you would be interested in working on, (3) links to relevant datasets (some starting points for finding datasets: Germanic low-resource languages/dialects – Blaschke et al. 2023 (web overview), regional languages of Italy – Ramponi 2024, Arabic dialects – Guellil et al. 2021) and/or details on what kind of data you would like to collect/annotate, and (4) what focus you are most interested in (modelling/engineering, linguistically motivated error analysis, etc. – you are very welcome to pitch specific research questions here). General reference: Zampieri et al. 2020. Level: BSc or MSc.

    • Analyzing dialect identification systems. How well can we automatically discern between different closely related language varieties, and do the features that are relevant to the success of an automatic classifier correlate with those that linguists would describe? You can decide on the language varieties to be included in the thesis, provided that high-quality, accessible corpora are available. (See above for starting points for finding datasets. If you prefer to work with a language not covered in these overviews, please contact us early on.) You can also decide which thematic focus/foci the thesis should have: (i) linguistic analysis of the data and classifier output; (ii) applying interpretability methods to the classifier output (in conjunction with focus i); (iii) comparing different kinds of input representations and ML architectures, including speech. The specific focus and level of detail will depend on your background and skills as well as the degree (BSc vs. MSc). Additional references: Gaman ea. 2020 and other VarDial shared task papers for literature on automatic dialect identification, Nerbonne ea. 2021 and Wieling & Nerbonne 2015 for introductions to dialectometry; Madsen ea. 2022 and Barredo Arrieta ea. 2020 for overviews of interpretability methods. Level: BSc or MSc.

    • Slot and Intent Detection for Low-Resource Dialects. Digital assistants are becoming wide-spread, yet current technology covers only a limited set of languages. How can we best do zero-shot transfer to low-resource language variants without standard orthography? Reference: van der Goot et al., 2021 and VarDial 2023 SID4LR. Create a new evaluation dataset of a low-resource language variant you speak, and investigate how to best transfer to it. Topics: Transfer Learning, Cross-linguality, Dataset annotation (Particularly suited for students interested in covering their own language or dialect not yet covered by existing systems including local dialects, e.g. Austrian, Low Saxon, Sardinian dialects or others). Level: MSc.

    • Creating a dialectal dependency treebank or POS-tagged corpus. Create a small Universal Dependencies (UD; de Marneffe et al. 2021) treebank in a dialect, regional language or other low-resource language that you are familiar with. For a less time-intensive project, it is also possible to only annotate part-of-speech (POS) tags (otherwise: POS tags + dependencies) and complement the project with something else. This project requires a strong interest in linguistics and syntax. You will need to read up on UD’s annotation guidelines and independently seek out relevant linguistic literature on your chosen language. You also evaluate parsers/POS taggers on your new dataset in a cross-lingual transfer set-up and, time permitting, you might also train your own parsers. Ideally, the project leads to contributing a new treebank to the UD project. Examples of similar corpora: Siewert et al. 2021, Hollenstein & Aepli 2014, Cassidy et al. 2022, Lusito & Maillard 2021. Tutorial for UD newcomers. Level: BSc or MSc.

    • Lexical Resources for Dialects. NLP for dialect languages is an intriguing area of research due to the lack of resources (low-resources languages) and lack of standardization (high variance). In this project, the goal is build dialect dictionaries by annotating words and phrases with respect to different linguistic properties including parts-of-speech, cases and grammatical genders. For this, we have collected raw data in several German dialects. It is therefore important that the student is familiar with one or more of the following dialects: Alemannic (Alemannisch), Palatinate (Pfälzisch), Frisian (Friesisch), Saterland Frisian (Saterfriesisch), Bavarian (Bairisch), Low German (Niederdeutsch/Plattdeutsch), Colognian (Kölsch). The second part consists of using these dictionaries to either 1) systematically investigate the linguistic dialect competence of pre-trained language models or 2) develop IR-specific resources (dialect stopword lists, dialect lemmatizers, dialect stemmers). This project does not need access to GPUs and is suitable for BSc and MSc students.

    • Transfer or translate: how to better work with dialectal data. Demands for generalizing NLP pipelines to dialectal data are on the rise. Given current LLMs trained in hundreds of languages, there are two common approaches. The first approach is to translate (or normalize) dialectal data to its mainstream counterpart and apply pipelines to the translated mainstream counterpart. Such an approach can benefit from the bigger amount of unannotated and annotated data in the mainstream variant but suffers from error propagation in the pipeline. The second transfer approach is to annotate a small amount of dialectal data and few-shot transfer (finetune) models on the dialect. This involves more dialectal annotation as well as collected unannotated dialectal data. Reference: Zampieri et al. 2020. For a BSc thesis, you would choose an NLP task (e.g., syntactic or semantic parsing, sentiment or stance detection, QA or summarization, etc.) and a specific dialect, compare performances of fewshot versus translation approaches quantitatively, and conduct a qualitative error analysis on the difficult cases. For MSc, the research needs to scale up either to multiple dialects (in the same or across different language families) or to multiple NLP tasks. Level: BSc or MSc.

    • To What Degree do LLMs Understand Bavarian Dialect Variants? In this project, we aim to comprehensively evaluate existing multilingual and German LLMs on Bavarian dialect variants. Your task is to come up with a set of evaluation criteria to test existing LLMs in zero-shot and few-shot manner, starting from existing benchmarks but with the aim to go beyond them, systematically comparing LLMs with traditional fine-tuning approaches, and finding out when and why certain methods struggle. Techniques: In-depth Evaluation, LLMs, Fine-tuning, Behavioral Testing. Level: MSc.

    • Language Modeling of Historical Non-Standard Language Documents. Digitalisation can provide access to valuable historical information, especially for non-standard languages and dialects. In this project, you test and build a prototype for digitalisation of historical documents using recent visual representation-based methods. The project includes: data gathering and annotation, model evaluation and improvement (e.g. by augmentation methods). References: Salesky et al., 2021, Borenstein et al., 2023. Level: MSc.

V2: High-Quality Information Extraction and Retrieval, Data-centric NLP

Selected research projects

  • Computational Job Market Analysis. Job postings are a rich resource to understand the dynamics of the labor market including which skills are demanded, which is also important for an educational viewpoint. Recently, the emerging line of work on computational job market analysis or NLP for human resources has started to provide data resources and models for automatic job posting analysis, such as the identification and extraction of skills. See references of MultiSkill project. For students interested in real-world applications, this theme provides multiple thesis projects including but not limited to: an in-depth analysis of existing data set and models, researching implicit skill extraction or cross-domain transfer learning to adapt skill and knowledge extraction to data sources other than job postings like patents or scientific articles. See references of MultiSkill project. See also Bhola et al., 2020 and Gnehm et al. 2021 and our own ESCOXLM-R model. Level: BSc or MSc.

  • Climate Change Insights through NLP. Climate change is a pressing issue internationally that is receiving more and more attention everyday. It is influencing regulations and decision-making in various parts of society such as politics, agriculture, business, and it is discussed extensively on social media. For students interested in real-world societal applications, this project aims to contribute insights on the discussion surrounding climate change on social media by examining discourse from a social media platform. The data will have to be collected (potentially from existing sources), cleaned, and analyzed using NLP techniques to examine various aspects or features of interest such as stance, sentiment, the extraction of key players, etc. References: Luo et al., 2020, Stede & Patz, 2021, Vaid et al., 2022. Level: BSc or MSc.

  • Better Benchmarks / Mining for Errors in Annotated Datasets. Benchmark datasets are essential in empirical research, but even widely-used annotated datasets contain mistakes, as annotators inevitably make mistakes (e.g. annotation inconsistencies). There are several lines of work in this direction. On the one site, annotation error detection methods provide a suite of methods to detect errors in existing datasets (cf. Klie et al. 2023, Weber & Plank 2023), including tools such as data maps (Swayamdipta et al. 2020). On the other side, there is work on inspecting existing datasets in revision efforts that exist for English NER in the past year (cf. Reiss et al. 2020, Rücker & Akbik 2023). The goal of projects on this theme can be:
    a) (MSc level) to investigate error detection methods in novel scenarios (new benchmarks, new applications, and/or create a new error detection dataset),
    b) (MSc or BSc level) extend revision efforts on NER to other languages. For the latter, for a BSc thesis, your task includes improving a benchmark dataset with iterations of sanity checks and revisions and comparing NLP models on the original versus revised versions. For MSc, you could extend either by incorporating Annotation Error Detection methods (see previous part) or conducting additional evaluations on multiple downstream NLP tasks.
    c) (MSc level) Checking the annotation consistency of non-standardized language data. Automatic methods for finding potential inconsistencies in annotations typically rely on consistent orthographies (e.g., detecting sequences that occur multiple times in a corpus but have received different annotations; Dickinson & Meurers 2003). When text is written in a language variety without a standardized orthography, such methods may no longer work well because of spelling differences between the repeated sequences. Your task is to extend such approaches to detect errors in existing datasets to be more robust to orthographic variation and/or to investigate how well annotation error detection methods that do not directly depend on orthographic matches work (cf. Klie et al. 2023). The target dataset would ideally be a dialectal dataset currently under development at the lab (this requires familiarity with German dialects and an interest in syntax).

  • Error Analysis of a BERT-based Search Engine: Multi-stage ranking has become a popular paradigm in information retrieval, this approach a fast first-stage ranker generates a candidate set of documents followed by a much slower re-ranker to refine the ranking (Nogueira et al. 2019). Prior work has shown that better candidate sets (higher recall) don’t necessarily translate to a better final ranking (Gao et al. 2021). The goal of this thesis is two-fold: First, we would like to perform an error analysis of linguistic triggers that cause this behavior. In the second part, the goal is to apply and interpret automatically generated explanations from tools such as DeepSHAP (Fernando et al. 2019) and LIME (Riberio et al. 2016). Basic knowledge in information retrieval is helpful, but not required. Level: B.Sc.

  • Injecting Lexical Similarity Signals into Neural Information Retrieval (IR) Models. Early IR models rely lexical signals (keyword matches) such as BM25, TF-IDF and the Query Likelihood Model (QLM) between queries and documents to determine their relevane. For a long time lexical retrieval models performed still very competitively, outperforming neural retrieval in domain-specific settings Thakur et al. (2021). Reranking with cross-encoders (CE) is arguable among the most widely used IR paradigms and frames relevance prediction as a sequence pair classification task where inputs are constructed through concatenating query-document pairs “[CLS] Query [SEP] Doc [SEP]”. In a recent work, Askari et al (2023) showed that CEs benefit from further including lexical input signals “[CLS] Query [SEP] BM25 [SEP] Doc [SEP]”. The goal of this thesis is to conduct a systematic evaluation of additional lexical signals (and combinations thereof) including, e.g., TF-IDF and QLM (reduced scope, BSc) and semantic signals obtained from semantic similarity models or LLMs (full scope, MSc). This thesis is suitable for students who do not have access large GPUs. Level: BSc or MSc.

  • Investigating Information Asymmetry on Wikipedia. Information asymmetry (IA) refers to the fact that the volume and type of information on the web varies between languages. For example, prior work by Kolbitsch and Maurer (2006) and Callahan et al. (2011) find that articles about locally famous people (“local heros”) are written in a more favorable way whereas articles about the same people written in other languages also discuss controversial aspects. As a result, monolingual search engines can produce culturally biased search results. The goal of this thesis is to investigate IA on Wikipedia. In the first part, the student is expected to devise an annotation schema that allows to systematically analyze IA and then use it to compare articles about “local heroes” in 2-3 different languages. The second part consists of delving deeper into measuring IA in an automated way. The student is expected to investigate different ways of applying unsupervised semantic similarity models (e.g. BERTScore proposed in Zhang et al. (2019)) or LLMs to identify IA. This thesis is suitable for BSc students (under a reduced scope) and MSc students. Level: BSc or MSc.

  • Do LLMs suffer from lexical biases in learning to rank (L2R)? LLMs are used ubiquitously in virtually all areas of NLP. This includes information retrieval (IR), where LLMs are used to, e.g., judge query-document pairs to predict relevance (see Fig. 6 in Yutao et al. (2023)). In the context of cross-lingual IR (CLIR), we previously showed that multilingual pre-trained language models trained on English data suffer from monolingual overfitting Litschko et al. (2023), i.e. they are biased towards predicting relevance based on keyword matches instead of semantic matches. We showed that bias can be mitigated by code-switching query and document tokens. The focus of this thesis is to (a) explore whether LLMs also suffer from monolingual overfitting and (b) if replacing random tokens with their translations improve their CLIR performance. This question can be investigated through the lens of prompt engineering, in-context L2R or instruction-tuned LLMs. Level: MSc.

  • V3: Natural Language Understanding, Semantic & Pragmatics, Computational Social Science

Selected research projects

  • Understanding Indirectness. Indirectness involves for example indirect answers to requests that do not explicitly contain answer clues like Yes, yeah or no. Example: Q: Do you wanna crash on the couch? A: I gotta go home sometime. Indirect answers are natural in human dialogue, but very difficult for a conversational AI system. This thesis project evolves around creating technology that can better deal with indirectness, for example by improving indirectness classifiers, creating novel datasets or integration of context, for example by modeling longer dialogue context. The exact project scope will be determined with the student. References: Louis et al., 2020, Damgaard et al., 2021 and Sanagavarapu et al., 2022, Yusupujiang & Ginzburg, 2023, Müller and Plank, 2024 Level: MSc (preferred), BSc (possibly).

V4: Human-centric Natural Language Understanding: Uncertainty, Perception, Cognition, Vision, Interpretability

Some general references for this section: References: Plank, 2016., Jensen and Plank, 2022, Plank, 2022 EMNLP

Selected research projects

  • In-context learning from human disagreement on subjective tasks. Aggregating annotations via majority vote could lead to ignoring the opinions of minority groups, especially on subjective tasks. Learning from individual annotators shows a better result on subjective classification tasks such as hate speech detection and emotion detection than from the majority vote (Davani et al., 2022). In this project, we want to investigate the potential of learning from individual annotators in an in-context learning setting. How can the model learn from the disagreement between annotators by instruction tuning and prompting? How do we design such instructions? References: Davani et al., 2022, Schick et al., 2021 and Mishra et al., 2022. Level: MSc.

  • Beyond the Multiple Choice Survey Question: Evaluating LLM’s opinion in realistic settings. Multiple choice questions play a crucial role in LLM evaluation studies. Given an instruction and example QA pairs, the LLM is asked to pick the preferred answer, such as the MMLU benchmark (Hendrycks et al. 2021). Other than objective tasks such as math questions, survey questions are also used to evaluate LM’s opinion (Santurkar et al. 2023). However, it is still unclear how well the survey question responses of the LM represent its behavior in real-world scenarios. In this work, we want to investigate how representative the survey question evaluation result is and develop an open-ended generation task for evaluating LLM in a realistic setting. Level: BSc, MSc.

  • Understanding disagreement in human-generated scores. Large collections of human evaluations on a scale (e.g., how concrete word X is on a scale from 1 to 5) are crucial for modeling various linguistic phenomena (e.g., figurative language use). These collections typically report an average score derived from numerous participants. However, this method of averaging can be highly deceptive, especially when individual scores vary significantly and the average score converges to a middle-range score. This project aims to explore the characteristics of words that achieve mid-scale average scores, examining their lexical attributes (like frequency), distributional properties (such as associations with other words), and behavioral and cognitive aspects (for instance, emotional connotations). This study will build on the existing work by Knupleš et al. (2023). Expected skills: regression analysis, cluster analysis, data visualization. Level: MSc.

How to apply for a BSc and MSc thesis project

Important information for LMU students: You need to apply for a MSc/BSc thesis project the latest three weeks before the thesis project registration date.

Deadlines for the winter semester 2024-2025:

  • MSc students apply before September 4, 2024
  • BSc students apply before September 14, 2024

Deadlines for the summer semester 2025:

  • MSc students apply before February 12, 2025
  • BSc students apply before February 26, 2025

To apply, please send your application material with subject “[BSc (or MSc) thesis project at MaiNLP - inquiry [Name and which semester]” to: thesisplank@cis.lmu.de

It should contain a single pdf with the following information:

  • CV, your study program, full grade transcript
  • Level: BSc or MSc thesis project
  • Which theme or concrete project interests you the most (optional: we are open to project proposals related to the research vectors or on-going research projects above). If you are interested in multiple, list your preferences for up to two (ranked list: first priority, second priority)
  • Languages you speak
  • Your favorite project so far, and why
  • Your knowledge and interest in data annotation, data analysis and machine learning/deep learning (including which toolkits you are familiar with)
  • Whether you have access to GPU resources (and which)
  • A term project report or your BSc thesis if you apply for a MSc thesis (optional)

Reach out if you have questions, using the email above.