At Taichung Green Museumbrary, two distinct sensibilities coexist under one roof: the visual intensity of an art museum and the narrative depth of a library. Architecturally, they are unified. Digitally, however, they have long spoken different languages.

Our collections include around 5,300 library records spanning art, design, literature, and everyday life, alongside thousands of artworks from the museum. Traditionally, these materials are accessed through keywords. Search for “spring,” and the system returns a gardening guide or an image of blooming flowers. Accurate, but limited.

What’s missing is what curators value most: serendipity, and the underlying narratives that connect seemingly unrelated works.

As an software engineer, I found myself questioning the premise. Perhaps the goal isn’t simply to store and retrieve data, but to build a bridge between structured information and human perception.

Could a system move beyond keywords—and instead connect works through mood, texture, or even metaphor?

At first, we treated this as a standard multimodal embedding problem. In theory, if images and text are mapped into the same vector space, they should align naturally.

In practice, they didn’t.

When the design team posed prompts like “the shape of melancholy,” the results were often unintentionally literal. A query like “detectives” and “hidden truths” returned books about communication skills, rather than anything resembling noir aesthetics or suspense.

This wasn’t a failure of the model—it was a semantic gap. To the system, an image of the ocean is simply “blue” and “water.” To us, it might evoke loss, memory, or the passage of time. The model can identify objects, but not always what they imply. It sees a flower, but not the joy of blooming. It detects geometric ice, but not the cold precision of logic. Without addressing this gap, even the most advanced models remain shallow.

So we changed our approach. Instead of relying solely on raw metadata, we introduced a large language model as an interpretive layer. Rather than embedding images and text directly, we first asked the AI to describe each work—like a critic would.

For every book and artwork, the system generates descriptions across three dimensions:

• Visual — What is present? Shapes, light, composition
• Style — Medium, technique, aesthetic language
• Affect — What does it evoke? Solitude, tension, stillness, celebration

Next, we “stitch” these descriptions together. Instead of just saving tags, we plug the AI-analyzed creator, concrete keywords, abstract concepts, and emotions into a standardized narrative template:

This work by [Name] is defined by its [Keywords] style. It delves into the concept of [Abstract].

Instead of storing isolated keywords, each item becomes a coherent description—a shared language that both literature and art can inhabit.

Once this interpretive layer was introduced, the system began to behave differently.

We could now curate using poetic prompts—and the system would respond meaningfully.

For a theme like “The Sound of Winter,” the prompt was: “What does ice remember?” The system associated visual features such as crystalline geometry and negative space with abstract ideas like logic, silence, and permanence.

The result was a pairing: a minimalist geometric artwork alongside a philosophical text exploring structures of reasoning.

This is the kind of fusion we were aiming for.

The library provides the “bones” (depth and philosophy), while the art gallery provides the “flesh” (visual impact and beauty). They aren’t just linked by keywords; they share a common language of ‘cool rationality’ that sparks a dialogue across different media.

Or, through the “scent of spring”—using cues like “morning mist” and “tender buds”—the system connects an ink wash painting with a science book about the origins of life. This is no longer just a search; it is an exploration of concepts.

Looking back, the role of the software engineer felt less like building a system, and more like translating between worlds.

We weren’t just processing data—we were attempting to encode intuition. The system is no longer a search tool. It becomes something more responsive—capable of engaging with ideas like stillness, weight, or ambiguity.

At Green Museumbrary, our goal is for the technology to disappear. When a visitor stands before the circular screen and, prompted by a question like “How high can mist rise?”, discovers an unexpected connection—that moment of recognition is the real outcome. That is where the system becomes meaningful.

Photo: Courtesy of YHLAA
Text by: 椎名貝波
Key Visual: 一杯中冰美