The question is how to index a visual database. Consider a visual database -- collection of drawings, three-dimensional models, scanned photographs, video, and text -- as a kind of modernmultimedia architectural sketchbook. It can be shared among a wide group of users with different purposes, and who may think about the contents in rather different ways. The connections -- perhaps hypertext -- among the entries may be complex and the organization difficult to comprehend. How then, to index the collection? Certainly traditional techniques -- looking for a concert hail -- built of concrete and glass -- in the 1970's in Utrecht and the architect's name is H* -- will help. But suppose we do not know so precisely what we are looking for? Might we appeal to the language of diagram? Can we add to our schemes for search and retrieval a diagrammatic index? We propose to try this idea. The paper describes our “computer as cocktail napkini system for recognizing and interpreting diagrams. It consists of a pen-based freehand sketching program that recognizes simple symbols the user has trained (such as lines, shapes, letters, etc.) and spatial arrangements of these symbols. A graphical search procedure finds occurrences of a drawn configuration of symbols in the pages of a sketchbook made using the program. By extending thepages of the sketchbook to include photos, drawings, and text in addition to diagrams, we can use this technique to find items whose diagrams match a drawn search configuration. The paper will demonstrate this prototype program and explore its use for indexing visualdatabases in architecture.