- A unIfied framework for multimodal content SEARCH

FP7 Logo


The aim of the I-SEARCH project is the development of the first search engine able to handle a wide range of specific types of multimedia and multimodal content (text, 2D image, sketch, video, 3D objects, audio and combination of the above), which can be used as queries and retrieve any available relevant content of any of the aforementioned types.

In order to achieve this goal the following main objectives should be fulfilled:

Research and development of an innovative Rich Unified Content Description (RUCoD).

RUCoD will consist of a multi-layered structure (from low-level to high-level descriptors), which will integrate content’s geometrical, topological, temporal, multisensory and multimodal information and meta-tags connected with the intrinsic properties of the content (static features such as shape, colour, texture, dimension, etc.), dynamic properties (temporal descriptors, how it behaves, in which activities it is normally used, who uses it, etc.), non-verbal expressive and emotional descriptors, social descriptors (how content is related to users, social/collaborative use of the content), content descriptors as for the behavior of the humans included in the (visual and vocal) content, descriptors for users’ behavior as for how content is intended to be elaborated and manipulated, individually or socially.

Further, novel multimodal annotation propagation algorithms will be developed, i.e. annotating information of one form/modality using information describing other forms/modalities of the same object.

Development of intelligent content interaction mechanisms so that only the content of interest will be delivered to the users.

This will be achieved by providing users with natural and expressive and multimodal interfaces as well as through personal and social-based relevance (including recommendation-based) feedback mechanisms. In this context, social and collaborative behavior of users interacting with the content will be exploited at best, which will help users to better express what they want to retrieve.

Provide a novel way for presentation of the multimodal data retrieved by the search engine, by utilizing visual analytics technologies.

Research will focus on exploiting the unified content representation for generating advanced reasoning algorithms to drive the visualisation of the I-SEARCH RuCoD compliant content. Visual Analytics will provide an analytical process for presenting the search results in the optimal way to aid the user in finding the result that optimally matches the query in a fast and efficient way.

  • Twitter
  • Facebook
  • LinkedIn
  • Share/Bookmark
Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.

© 2010 I-SEARCH Project Consortium