Large Language Models and their Ability to Capture Meaning in Unique Vocabularies

Reserach for my Master’s in Computer Science, Vector Institute Collaborative Specialization in AI
@ Western University, London, Ontario, Canada
Supervisor: Professor Dan Lizotte
Associate Professor, Department of Computer Science, Westen University

Overview

For my Master's research, I am developing interpretability techniques to assess how Large Language Models (LLMS) capture the meaning of words from highly unusual and unique vocabularies. These unique vocabularies include:

Reddit communities (and other social media) that utilize memes, slang, and colloquial terms, which constantly evolve as internet culture shifts over time
Industries that employ specialized terminologies, with some terms having different meanings outside of those industries.

LLMs I'm Experimenting With

Falcon-7B

QLoRA-Alpaca-13B

GPT-NeoX-20B

Experimental Design

I will be developing 3 experiments to discover and evaluate what information LLMs captures when inputting unique vocabulary. Refer to the diagram below to get a high-level overview of how each experiment evaluates different layers of an LLM and the details of each experiment. (All experimental designs shown below were designed and proposed by myself and approved by my professor).

Go Home

Large Language Models and their Ability to Capture Meaning in Unique Vocabularies

Reserach for my Master’s in Computer Science, Vector Institute Collaborative Specialization in AI@ Western University, London, Ontario, CanadaSupervisor: Professor Dan LizotteAssociate Professor, Department of Computer Science, Westen University

Overview

LLMs I'm Experimenting With

Experimental Design

Reserach for my Master’s in Computer Science, Vector Institute Collaborative Specialization in AI
@ Western University, London, Ontario, Canada
Supervisor: Professor Dan Lizotte
Associate Professor, Department of Computer Science, Westen University