lloomr: An R Tool for Automatic Concept Induction from Text Using Large Language Models

Introducing the lloomr project, an R implementation of the LLooM algorithm that automatically discovers interpretable concept structures from large text corpora, supporting concept scoring, single-label classification, and visual analysis.

• Original Author/Maintainer: Jan Zilinsky
• Source Platform: GitHub
• Original Title: lloomr: Concept Induction from Text with Large Language Models
• Original Link: https://github.com/zilinskyjan/lloomr
• Release Time: 2024 (based on CHI 2024 paper)

---

When dealing with large-scale text data, researchers often face a core challenge: how to extract meaningful and interpretable concept structures from unstructured text collections? Traditional methods often rely on manual coding or pre-defined classification systems, which are not only time-consuming and labor-intensive but also struggle to capture emergent implicit patterns in the data.

The LLooM (Large Language Model-based concept induction) algorithm was developed to address this problem. It was first proposed by Michelle Lam et al. at the CHI 2024 conference and has a Python implementation. The lloomr project is an R port of this algorithm, developed and maintained by Jan Zilinsky, allowing R users to seamlessly use this powerful concept induction tool.

---

lloomr uses a six-stage pipeline design to transform raw text into a structured concept system:

First, the system uses a large language model to distill each piece of raw text into key points (bullets). This step compresses lengthy documents into manageable core information fragments while preserving the semantic essence of the original text.

Next, the system vectorizes the distilled text, then uses UMAP dimensionality reduction and HDBSCAN clustering algorithms to group semantically similar text fragments. This stage does not require predefining the number of categories; the algorithm automatically discovers naturally occurring topic groups in the data.

This is the core step of the entire process. The system uses a large language model to generate concept proposals for each cluster group, including a concept name and a one-sentence inclusion criterion. Unlike traditional topic modeling, the concepts generated here have clear semantic boundaries and interpretability.

The generated concepts need to be screened and optimized. Users can remove redundant concepts, merge similar concepts, or select the most relevant subset. This human-machine collaboration step ensures the quality and practicality of the final concept system.