Image Database
Introduction.
This document outlines some initial ideas on how an image database might be written. Such a system would not be specific to images and would probably be better referred to as an 'object' or 'blob database'. I will stick with 'image database' in this document to avoid confusion.
There are currently many images databases available on the market. Ranging from simple web based photo albums to full scale commercial document management systems used by business such as national newspapers. This project aims to create a simple, yet extendible system, that is suitable for all categories of user and can be that can be customised to specific needs. Primarily the system should be easy to use. Many systems are abandoned because their use is seen as a chore. Ease of use however should not preclude extensibility. This system aims to define a core set of tools that meets the use of most users. More advanced users will be able to build upon the simple base and customise to their own needs.
Meta Data.
Meta data is the data that tells us something about the data we are storing. Commonly stored meta data would be such things as 'Author', 'Creation Date', 'Copyright Information' etc.
Meta data is the key to the success of any project of this nature. A lack of meta data or poorly defined or stored meta data leads to poor searching and organisation of the data. If you can't find and sort your images then you might as well have just stored them in a directory on your hard disk or in a pile on the floor. Good meta data on the other hand allows you to classify your images, search for particular images and tell other people about the images you have.
One of the principal problems with a lot of images database systems is not their inability to store meta data but that they make it difficult to enter and customise. Consequently many images are stored in the world but few are stored with sufficient meta data. To a certain extent, the problems with meta data entry and upkeep are down to the user ( laziness is a very large problem in many projects ) and as such cannot be solved by any computer system. However, on one hand the system should make the meta data simple to understand and easy to enter, on the other hand the meta data should not be so simple that it makes it of little use to the system or external users.
Storing Meta Data.
A major thrust of computing in recent years has been the development of XML and its associated standards. XML allows data interchange and storage in a well defined way, free of proprietary file formats. Storing meta data in XML makes it a lot more use to external users and increases the range of tools that can be used to process the data. However, while XML defines the syntax of storage it does not define what is valid within that syntax. Other standards such as XML Schema go on to define what the structure of an XML document looks like. An image database system could use straight XML files to describe the meta data for each image, without specifying how the meta data is stored and processed. Such a system however would be difficult to create and use. A solution to this is to constrain how the data is stored by using the W3C's RDF standard. This is a scheme designed specifically for storing meta data and forms the foundation of the W3C's semantic web schema. Quoting from the W3C's RDF primer :-
"RDF is based on the idea of identifying things using Web identifiers (called Uniform Resource Identifiers, or URIs), and describing resources in terms of simple properties and property values. This enables RDF to represent simple statements about resources as a graph of nodes and arcs representing the resources, and their properties and values."
This combination of resource, property and value is known as a triple. A triple being a piece of information about something made up of ( in the W3C's nomenclature ) a Subject, Predicate and an Object. The subject is the thing we are providing information about, in our case an image. The predicate is a description of a property that belongs to the subject, for example author. The object is the data corresponding to the predicate, in this case the text representing the authors name. So, an RDF triple telling you that a certain image has me as the creator would look like :-
<http://www.mondaymorning.org/some_image.jpg>, <http://purl.org/dc/elements/1.1/creator>, "Richard Baker"
Here I have used the Dublin Core definition for a "creator". As can be seen RDF uses URI's to identify things, giving it a useful name spaced system that stop collision of identifiers.
The W3C also has a definition for an RDF schema language that defines elements that can be used to store data, and a way of serialising RDF to an XML file. Thus RDF gives us a simple ( to the user ) key, value pair way of storing meta data about an image that can be written to an XML file. The schema language gives us a way of defining what key, value pairs we want to store for a certain class of image. See the W3C RDF Primer for an introduction to RDF.
Because RDF is a well known standard there is plenty of software support for processing RDF/XML files, also organisations such as Dublin Core define schemas to provide definitions for common data entries ( such as creator, creation date etc. ). Since the data is stored in XML and the schemas are published, meta data can be exchanged much more easily between systems. In addition RDF reduces the scope for what is permissible when compared to vanilla XML files, so designing and implementing an image database system becomes a lot less complex.
Data Organisation.
A complete entry in this system will be an image and zero or more RDF data statements describing it. The image may be stored in a disk file, database or on a web server. Similarly, the RDF data may be stored in an XML file on disk, in an RDF database or on a web server. Storage location is not important so long as a user can usefully work on the image and its RDF data. To usefully work on these pairs the user will need to see a logical layout of the images so that he/she may navigate to them. A simple and well known way of doing this is to layout the image/data pairs in a directory structure. It is important at this point to emphasise that there is no restriction on how the data is stored internally, just on how it appears to the user. Since directory structures are well known to computers users this is an efficient way of working with the data.
This system will base its logical data storage hierarchy on the Unix file system. That is there will be a root directory '/' at the top of the tree. This may contain data files and other directories which may in turn contain files and other directories and so on. It will be possible to 'mount' an image database file system on to a directory in the hierarchy. This is analogous to the Unix file system mount and will allow a client of the system to present a hierarchy to the users, that while appearing to be a single data source may in fact be drawn from several different sources.
It should also be possible to store an RDF meta data file about each directory as well so that a user may describe the directories as well as the files within them.
Incremental Client Development.
While the range of features such a system could usefully include is very large it should be possible to design and write a fairly simple GUI client with a minimum of features as an initial technology demonstrator. Once an initial client has been designed and shown to meet the requirements of an image database system it could then be incrementally extended with the further desired features. With this in mind an initial client is proposed with the following features :-
- Native GUI interface. By writing a native binary program we present a client that is both faster ( than a web based system ) and easier to use than a non native system. Using a cross platform development toolkit such as Wx Widgets the client can be developed to run on several different platforms with the minimum of porting.
- Abstracted file system. By writing the client with an abstracted file system internally ( borrowing heavily from the concept of the Unix VFS system ) it will be possible to add different data providers more efficiently during later development.
- Basic local file system data provider. Once the abstracted file system is in place the initial version of the client will be provided with support for a local file system data provider, i.e. it will store the images and meta data in a local file system directory. This will be the quickest data provider to develop.
- Minimal RDF editor. Allow a user to store key value pairs about an object. The editor should not rely on the user knowing how to create URI's for the RDF data. Later development will add support for RDF Schemas so that a user can select a desired schema and fill in the values defined by that schema.
- Plug-in architecture for displaying and when required manipulating object types. Initially only an image plug-in will be written so that the client can display a small set of image types in the GUI. As further development continues extra plug-ins will be created for other object types ( e.g. sound, video, PDF etc. ).
- Basic searching. RDF application libraries provide basic searching across RDF data ( e.g RDQL ). This can be incorporated into the client to allow a basic initial search system. Future development will be able to add such functionality as free text search etc.
Web Output.
Once the client has been developed a web output module can be developed to provide greater access to the image database. Using the code developed for the abstract file system and the various drivers for data providers a web output system can be quickly developed that has access to the image database. By using a template system or XLST type transformation on the meta data ( both schemes have lots of existing software support ) pages can be quickly generated.
Network Server.
A system that uses the local file system as the data store will be relatively quick to develop, but it will not provide the sort of features required in an enterprise database system. The ultimate aim of this project would be to develop a network based data provider. Since the client has already been created with the concept of different data providers this will not change and for the most part the user will see no difference between the network provider and a local file system provider until they wish to use the extended features. It is by keeping this one existing client program that the system maintains a simple single interface for the users.
A network based provider would has such features as :-
- TCP/IP access allowing users spread across a wide geographical area to access the data.
- A knowledge of user identities and groups of users.
- Access control based on user and group permissions.
- Storage of RDF meta data in a purpose built RDF database. This would allow for a more robust and efficient way of storing the meta data.
- Manageable image data storage. Server storage of the image data should be configured in such a way that the administrator can increase/decrease the storage as required.
- Organised backup. A network server would provide support for backing up of the data in a standard way. Allowing full and incremental backups. Ideally such a system should be able to support commercial backup system.
- Web Access component. A network based data provider should allow live access to the data for those users who do not have access to the client program, in this way an institution will be able to share its data with a wider audience.