A "Transderivational" Search Engine for Creative Analogy Generation in Mixed-Media Design

Dinh, Huong
Stevens Institute of Technology

This project will develop a transderivational search engine based on the neurological condition known as synaesthesia in which two or more senses are crossed (e.g., when seeing a color causes one to hear a sound) to help people to discover connections between text, 1D audio, 2D image, 3D geometry and 4D motion data. The project is inspired by the ability of artists and designers to find analogies between diverse artifacts and bring them together to compose a coherent and novel narrative.

The intellectual merit of this research is the development of matching algorithms that suggest analogies across different media forms by looking at structural similarity within media content. The result will be a transformative technology at the intersection of art, computer graphics, machine learning, cognitive psychology, and human-computer interaction (HCI). Transderivational search will enhance the synaesthetic effect in analogy generation and will naturally lend itself to a wide range of brainstorming pursuits. Finding analogies between media of different forms (e.g., audio and 3D shapes) has not been explored, nor has there been much focus on non-literal search engines. Literal searches rely only on explicit meaning (e.g., the word “three” and an image of the number 3) and categorization to determine similarity. Instead, this project will compare media samples by looking for structural similarity using analytical approaches such as statistical shape distributions, frequency analysis, and machine learning techniques to discover relationships between mixed- (multi-dimensional) media samples.

The technical impacts of this research are in advancing artificial intelligence through transderivational search (essential to language and cognitive processing) and in opening up new research questions on search technology. The educational impacts are in drawing more women and minorities into CS and improving retention in CS programs by showing the relevance of search technology to creative design and to multimedia management. The transderivational search tools will be used by students in introductory level CS courses to build basic media management software. Transderivational search can also serve as a testbed for exploring algorithms in high level CS courses on machine learning, computer vision and graphics.

Preliminary Results:
Over the past year we have been developing algorithms to compare vector fields. We believe that vector fields are a key meta-representation for different media forms. For example, vector fields can be extracted from video data through optical flow and particle tracking algorithms. Vector fields may also be extracted from certain types of images such as pen and ink drawings (where contour lines follow the surface's principle directions) or from impressionist style paintings where brush strokes (which may be described by vectors) are used to convey motion. Our preliminary results on finding patterns in vector fields are described in the following papers:

Global matching: http://www.cs.stevens.edu/~quynh/papers/vecfld_globalmatch.pdf
Local matching: http://www.cs.stevens.edu/~quynh/papers/vecfld_localmatch.pdf

Ongoing Work:
This summer, we are working with undergraduate student Victoria Petite in the Art & Technology program to develop surveys for understanding how analogies across images and video are perceived. An ongoing blog of her work on this project is at: http://vertigomotel.blogspot.com/. If you are aware of related work in this area from the graphics, arts, or cognitive science community, please email me.