EP90 - TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documents

Download the paper - Read the paper on Hugging Face

Charlie: Welcome to episode 90 of Paper Brief, where we dive deep into the cutting-edge of ML research. I’m Charlie, your host, and today I’m joined by Clio, an expert in machine learning and natural language processing. Today we’re unpacking the paper ‘TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documents’. So Clio, can you kick us off by telling us why explainability is so crucial for large language models?

Clio: Absolutely. Large language models are becoming pivotal in countless applications due to their highly accurate and coherent responses. However, their complexity makes them black-box models, and there’s an increasing need for post-hoc explainability. Shapley values have been key, but scaling them for models with long input contexts and generated output sequences is a real challenge.

Charlie: They sure have a wide reach. But what makes these models so versatile and yet so elusive when it comes to transparency?

Clio: They’re versatile thanks to their ability to ingest large contexts and excel in various text generation tasks. This means they can be used in both academic and commercial domains without needing specialized models. However, the same sophistication that allows them to tackle so many tasks makes it difficult to explain their internal mechanisms.

Charlie: Interesting. And when we talk about explanations, we often hear about ‘understandability’ and ‘faithfulness’. Could you shed some light on these terms for our audience?

Clio: Of course, ‘understandability’ refers to how easily someone can grasp an explanation, while ‘faithfulness’ measures how accurately an explanation mirrors the model’s decision-making process. It can be quite subjective, but it’s crucial to strike the right balance to make explanations valuable.

Charlie: Now, let’s dive into the paper’s big contribution—TextGenSHAP. What sets it apart from previous methods like Shapley values?

Clio: TextGenSHAP is designed to work with the scale of large language models, focusing on long inputs and complex tasks. It notably speeds up the explanation process from hours to just minutes or even seconds for document-level explanations. This makes it practical for use in real-time applications.

Charlie: A considerable upgrade, indeed. How does it change the game for tasks like question answering with long documents?

Clio: By providing real-time Shapley values, TextGenSHAP helps to understand and improve long-document question answering. It locates the important words and sentences, ultimately improving the accuracy of the responses generated by these models.

Charlie: And how does it compare to other post-hoc explainability methods out there?

Clio: Many existing methods like LIME or Integrated Gradients are great, but they’re primarily designed for simpler tasks and smaller data. TextGenSHAP addresses the needs of the NLP field with its ability to handle large output spaces that come with free-form text generation.

Charlie: This all sounds super promising for the future of NLP. Thanks for breaking it down for us, Clio.

Clio: My pleasure! It’s an exciting time for machine learning, and explaining these complex models is key to their wider adoption and responsible use.