This is a file in the archives of the Stanford Encyclopedia of Philosophy. 
version history

Stanford Encyclopedia of PhilosophyA  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z

last substantive content change

Challenging a longstanding prejudice against diagrammatic representation, those working on multimodal reasoning have taken different kinds of approaches which we may categorize into three distinct groups. One branch of research can be found in philosophy of mind and cognitive science. Since the limits of linguistic forms are clear to those who have been working on mental representation and reasoning, some philosophers and cognitive scientists have embraced this new direction of multimodal reasoning with enthusiasm and have explored human reasoning and mental representation involving nonlinguistic forms [Cummins 1996, Chandrasekaran et al. 1995]. Another strand of work on diagrammatic reasoning shows that there is no intrinsic difference between symbolic and diagrammatic systems as far as their logical status goes. Some logicians have presented case studies to prove that diagrammatic systems can be sound and complete in the same sense as symbolic systems. This type of result directly refuted a widelyheld assumption that diagrams are inherently misleading, and abolished theoretical objections to diagrams being used in proofs [Shin 1994,Hammer 1995]. A third direction in multimodal reasoning has been taken by computer scientists, whose interest is much more practical than those of the other groups. Not so surprisingly, those working in many areas in computer science — for example, knowledge representation, systems design, visual programming, GUI design, and so on — found new and exciting opportunities in this new concept of ‘heterogeneous system’ and have implemented diagrammatic representations in their research areas.
We have the following goals for this entry. First of all, we would like to acquaint the reader with the details of some specific diagrammatic systems. At the same time, the entry will address theoretical issues, by exploring the nature of diagrammatic representation and reasoning in terms of expressive power and correctness. The case study presented below will not only satisfy our first goal but also provide us with solid material for the more theoretical and general discussion in the third section. As mentioned above, the topic of diagrams has attracted much attention with important results from many different research areas. Hence, our fourth section aims to introduce various approaches to diagrammatic reasoning taken in different areas.
For further discussion, we need to clarify two related but distinct uses of the word ‘diagram’: diagram as internal mental representation and diagram as external representation. The following quotation from Chandrasekaran et al. [1995, p. xvii] succinctly sums up the distinction between internal versus external diagrammatic representations:
As we will see below, logicians focus on external diagrammatic systems, the imagery debate among philosophers of mind and cognitive scientists is mainly about internal diagrams, and research on the cognitive role of diagrams touches on both forms.
These existing diagrams have inspired those researchers who have recently drawn our attention to multimodal representation. Logicians who participate in the project have explored the subject in two distinct ways. First, their interest has focused exclusively on externallydrawn representation systems, as opposed to internal mental representations. Second, their aim has been to establish the logical status of a system, rather than to explain its heuristic power, by testing the correctness and the expressive power of selective representation systems. If a system fails to justify its soundness or if its expressive power is too limited, a logician's interest in that language will fade [Sowa 1984, Shin 2002].
In this section, we examine the historical development of Euler and Venn diagrams as a case study to illustrate the following aspects: First, this process will show us how one mathematician's simple intuition about diagramming syllogistic reasoning has gradually been developed into a formal representation system. Second, we will observe different emphases given to different stages of extension and modification of a diagrammatic system. Thirdly and relatedly, this historical development illustrates an interesting tension and tradeoff between the expressive power and visual clarity of diagrammatic systems. Most importantly, the reader will witness logicians tackle the issue of whether there is any intrinsic reason that sentential systems, but not diagrammatic systems, could provide us with rigorous proofs, and their success in answering this question in the negative.
Hence, the reader will not be surprised by the following conclusion drawn by Barwise and Etchemendy, the first logicians to launch an inquiry into diagrammatic proofs in logic,
there is no principled distinction between inference formalisms that use text and those that use diagrams. One can have rigorous, logically sound (and complete) formal systems based on diagrams. (Barwise & Etchemendy [1995], p. 214.)
This conviction was necessary for the birth of their innovative computer program Hyperproof, which adopts both firstorder languages and diagrams (in a multimodal system) to teach elementary logic courses [Barwise & Etchemendy 1994].
Leonhard Euler, an 18th century mathematician, adopted closed curves to illustrate syllogistic reasoning [Euler 1768]. The four kinds of categorical sentences are represented by him as shown in Figure 1.
Figure 1: Euler Diagrams
For the two universal statements, the system adopts spatial relations among circles in an intuitive way: If the circle labelled ‘A’ is included in the circle labelled ‘B,’ then the diagram represents the information that all A is B. If there is no overlapping part between two circles, then the diagram conveys the information that no A is B.
This representation is governed by the following convention:^{[2]}
Every object x in the domain is assigned a unique location, say l(x), in the plane such that l(x) is in region R if and only if x is a member of the set that the region R represents.
The power of this representation lies in the fact that an object being a member of a set is easily conceptualized as the object falling inside the set, just as locations on the page are thought of as falling inside or outside drawn circles. The system's power also lies in the fact that no additional conventions are needed to establish the meanings of diagrams involving more than one circle: relationships holding among sets are asserted by means of the same relationships holding among the circles representing them. The representations of the two universal statements, ‘All Aare B’ and ‘No A is B,’ illustrate this strength of the system.
Moving on to two existential statements, this clarity is not preserved. Euler justifies the diagram of “Some A is B” saying that we can infer visually that something in A is also contained in B since part of area A is contained in area B^{.[3]} Obviously, Euler himself believed that the same kind of visual containment relation among areas can be used in this case as well as in the case of universal statements. However, Euler's belief is not correct and this representation raises a damaging ambiguity. In this diagram, not only is part of circle A contained in area B (as Euler describes), but the following are true: (i) part of circle B is contained in area A (ii) part of circle A is not contained in circle B (iii) part of circle B is not contained in circle A. That is, the third diagram could be read off as “Some B is A,” “Some A is not B,” and “Some B is not A” as well as “Some A is B.” In order to avoid this ambiguity, we need to set up several more conventions.^{[4]}
Euler's own examples nicely illustrate the strengths and weaknesses of his diagrammatic system.
Example 1. All A are B. All C are A. Therefore, all C are B.
Example 2. No A is B. All C are B. Therefore, no C is A.
In both examples, the reader can easily infer the conclusion, and this illustrates visually powerful features of Euler diagrams. However, when existential statements are represented, things become more complicated, as explained above. For instance:
Example 3. No A is B. Some C is A. Therefore, Some C is not B.
No single diagram can represent the two premises, since the relationship between sets B and C cannot be fully specified in one single diagram. Instead, Euler suggests the following three possible cases:
Euler claims that the proposition ‘Some C is not B’ can be read off from all these diagrams. However, it is far from being visually clear how the first two cases lead a user to reading off this proposition, since a user might read off “No C is B” from case 1 and “All B is C” from case 2.
Hence, the representation of existential statements not only obscures the visual clarity of Euler Circles but also raises serious interpretational problems for the system. Euler himself seemed to recognize this potential problem and introduced a new syntactic device, ‘*’ (representing nonemptiness) as an attempt to repair this flaw. (Letter 105)
However, a more serious drawback is found when this system fails to represent certain compatible (that is, consistent) pieces of information in a single diagram. For example, Euler's system prevents us from drawing a single diagram representing the following pairs of statements: (i) “All A are B” and “No A is B” (which are consistent if A is an empty set). (ii) “All A are B” and “All B are A” (which are consistent when A = B). (iii) “Some A is B” and “All A are B”. (Suppose we drew an Euler diagram for the former proposition and try to add a new compatible piece of information, i.e., the latter, to this existing diagram.) This shortcoming is closely related to Venn's motivation for his own diagrammatic system (see Section 3.1 for other shortcomings of Euler's system).
Venn's criticism of Euler Circles is summarized in the following passage:
The weak point in this [Euler diagrams], and in all similar schemes, consists in the fact that they only illustrate in strictness the actual relation of classes to each other, rather than the imperfect knowledge of these relations which we may possess, or may wish to convey by means of the proposition. [Venn 1881, p. 510.]
Because of its strictness, Euler's system sometimes fails in representing consistent pieces of information in a single diagram, as shown above. In addition to this expressive limitation, Euler's system also suffers other kinds of expressive limitations with respect to nonempty sets, due to topological restrictions on plane figures (see Section 3.1).
Venn's new system [1881] was to overcome these expressive limitations so that partial information can be represented. The solution was his idea of ‘primary diagrams’. A primary diagram represents all the possible settheoretic relations between a number of sets, without making any existential commitments about them. For example, Figure 2 shows the primary diagram about sets A and B.
Figure 2: Venn's Primary diagrams
According to Venn's system, this diagram does not convey any specific information about the relation between these two sets. This is the major difference between Euler and Venn diagrams.
For the representation of universal statements, unlike the visually clear spatial containment relations in the case of Euler diagrams, Venn's solution is ‘to shade them [the appropriate areas] out’ ([Venn 1881], p. 122). By using this syntactic device, we obtain diagrams for universal statements as shown in Figure 3.
Figure 3: Venn's shading
Venn's choice of shading might not be absolutely arbitrary in that a shading could be interpreted as a visualization of set emptiness. However, it should be noted that a shading is a new syntactic device which Euler did not use. This revision gave flexibility to the system so that certain compatible pieces of information may be represented in a single diagram. In the following, the diagram on the left combines two pieces of information, “All A are B” and “No A is B,” to visually convey the information “Nothing is A.” The diagram below, representing both “All A are B” and “All B are A,” clearly shows that A is the same as B:
In fact, using primary diagrams also avoids some other expressivity problems (to do with spatial properties of diagram objects) discussed below, in Section 3. Surprisingly, Venn was silent about the representation of existential statements, which was another difficulty of Euler diagrams. We can only imagine that Venn might have introduced another kind of a syntactic object representing existential commitment. This is what Charles Peirce did about twenty years later.
Peirce points out that Venn's system has no way of representing the following kinds of information: existential statements, disjunctive information, probabilities, and relations. Peirce aimed to extend Venn's system in expressive power with respect to the first two kinds of propositions, i.e., existential and disjunctive statements. This extension was completed by means of the following three devices. (i) Replace Venn's shading representing emptiness with a new symbol, ‘o’. (ii) Introduce a symbol ‘x’ for existential import. (iii) For disjunctive information, introduce a linear symbol ‘’ which connects ‘o’ and ‘x’ symbols.
For example, Figure 4 represents the statement, ‘All A are B or some A is B’, which neither Euler's nor Venn's system can represent in a single diagram.
Figure 4: A Peirce diagram
The reason that Peirce replaced Venn's shading for emptiness with the symbol ‘o’ seems to be obvious: It would not be easy to connect shadings or shadings and x's in order to represent disjunctive information. In this way, Peirce increased the expressive power of the system, but this change was not without its costs.
For example, the following diagram represents the proposition ‘ Either all A are B and some A is B, or no A is B and some B is not A’:
Reading off this diagram requires more than reading off visual containment among circles (as in Euler diagrams) or shadings (as in Venn diagrams), but also requires extra conventions for reading combinations of the symbols ‘o,’ ‘x,’ and lines. Peirce's new conventions increased the expressive power of single diagrams, but the arbitrariness of its conventions and more confusing representations (for example, the above diagram) sacrificed the visual clarity which Euler's original system enjoys. At this point, Peirce himself confesses that ‘there is a great complexity in the expression that is essential to the meaning’ ([Peirce 1933], 4.365). Thus, when Peirce's revision was completed, most of Euler's original ideas about visualization were lost, except that a geometrical object (the circle) is used to represent (possibly empty) sets.
Another important contribution Peirce made to the study of diagrams starts with the following remark:
‘Rule’ is here used in the sense in which we speak of the ‘rules’ of algebra; that is, as a permission under strictly defined conditions. ([Peirce 1933], 4.361.)
Peirce was probably the first person to discuss rules of transformation in a nonsentential representation system. In the same way that the rules of algebra tell us which transformations of symbols are permitted and which are not, so should the rules of diagram manipulation. Some of Pierce's six rules needed more clarification and turn out to be incomplete — a problem which Peirce himself anticipated. However, more importantly, Peirce did not have any theoretical tool — a clear distinction between syntax and semantics — to convince the reader that each rule is correct or to determine whether more rules are needed. That is, his important intuition (that there could be transformation rules for diagrams) remained to be justified.
In [1994], Shin follows up Peirce's work in two directions. One is to improve Peirce's version of Venn diagrams, and the other is to prove the soundness and the completeness of this revised system.
Shin's work alters Peirce's modifications of Venn diagrams to achieve an increase in expressive power without such a severe loss of visual clarity. This revision is made in two stages: (i) VennI: retains Venn's shadings (for emptiness), Peirce's ‘x’ (for existential import) and Peirce's connecting line between x's (for disjunctive information). (ii) VennII: This system, which is proven to be logically equivalent to monadic predicate logic, is the same as VennI except that a connecting line between diagrams is newly introduced to display disjunctive information.
Returning to one of Euler's examples we will see the contrast among these different versions clearly:
Example 3. No A is B. Some C is A. Therefore, Some C is not B.
Euler admits that no single Euler diagram can be drawn to represent the premises, but that three possible cases must be drawn. Venn's system is silent about existential statements. Now, Peirce's and Shin's systems represent these two premises in a single diagram as follows:
In the case of Shin's diagram, Venn's shading convention for emptiness, as opposed to Peirce's ‘o’, much more naturally leads the reader to the inference “Some C is not B” than in the case of Peirce's diagram.
However, VennI cannot express disjunctive information between universal statements or between universal and existential statements. Retaining VennI's expressive power, VennII allows diagrams to be connected by a line. Peirce's confusing looking diagram above is equivalent to the following VennII diagram:
In addition to this revision, Shin [1994] presented each of these two systems as a standard formal representation system equipped with its own syntax and semantics. The syntax tells us which diagrams are acceptable, that is, which are wellformed, and which manipulations are permissible in each system. The semantics defines logical consequences among diagrams. Using these tools, it is proven that the systems are sound and complete, in the same sense that some symbolic logics are.
This approach has posed a fundamental challenge to some of the assumptions held about representation systems. Since the development of modern logic, important concepts, e.g., syntax, semantics, inference, logical consequence, validity, and completeness, have been applied to sentential representation systems only. However, none of these turned out to be intrinsic to these traditional symbolic logics only. For any representation system, whether it is sentential or diagrammatic, we can discuss two levels, a syntactic and a semantic level. What inference rules tell us is how to manipulate a given unit, whether symbolic or diagrammatic, to another. The definition of logical consequence is also free from any specific form of a representation system. The same argument goes for the soundness and the completeness proofs. When a system is proven to be sound, we should be able to adopt it in proofs. In fact, much current research explores the use of diagrams in automated theorem proving (see Section 4 and [BarkerPlummer & Bailin 1997, Jamnik et al. 1999]).
It is interesting and important to notice that the gradual changes made from Euler Circles through to Shin's systems share one common theme: to increase both the expressive and logical power of the system so that it is sound, complete, and logically equivalent to monadic predicate logic. The main revision from Euler to Venn diagrams, introducing primary diagrams, allows us to represent partial knowledge about relations between sets. The extension from Venn to Peirce diagrams is made so that existential and disjunctive information may be represented more effectively.
Both Venn and Peirce adopted the same kind of solution in order to achieve these improvements: to introduce new syntactic objects, that is, shadings by Venn, and x's, o's, and lines by Peirce. However, on the negative side, these revised systems suffer from a loss of visual clarity, as seen above, mainly because of the introduction of more arbitrary conventions. The modifications from Peirce to Shin diagrams concentrate on restoring visual clarity, but without loss of expressive power.
Hammer and Shin take a different path from these revisions: To revive Euler's homomorphic relation between circles and sets — containment among circles represents the subset relation among sets, and nonoverlapping of regions represents the disjoint relation — and at the same time, to adopt Venn's primary diagrams by default. On the other hand, this revised Euler system is not a selfsufficient tool for syllogistic reasoning, since it cannot represent existential statements. For more details of this revised system, refer to [Hammer & Shin 1998].
This case study raises an interesting question for further research on diagrammatic reasoning. Throughout the different developments of Euler diagrams, increasing its expressive power and enhancing its visual clarity seem to be complementary to each other. Depending on purposes, we need to give priority to one over the other. Hammer and Shin's alternative system provides a simple model for the development of other efficient nonsentential representational systems, a topic that has been receiving increasing attention in computer science and cognitive science.
A particular distinguishing feature of diagrams is that they obey certain “nomic” or “intrinsic” constraints due to their use of plane surfaces as a medium of representation. The idea is that sentential languages are based on acoustic signals which are sequential in nature, and so must have a compensatingly complex syntax in order to express certain relationships  whereas diagrams, being twodimensional, are able to display some relationships without the intervention of a complex syntax [Stenning & Lemon 2001]. Diagrams exploit this possibility — the use of spatial relations to represent other relations. The question is; how well can spatial relations and objects represent other (possibly more abstract) objects and relations?
Logical reasoning with diagrams is often carried out in virtue of their depiction of all possible models of a situation, up to topological equivalence of the diagrams (this, of course, depends on the particular diagrammatic system in use). A single diagram is often an abstraction over a class of situations, and once a suitable diagram has been constructed, inferences can simply be read off the representation without any further manipulation. In some diagrammatic systems (e.g., Euler Circles) inference is carried out by constructing diagrams correctly and reading information off them. The complexity of using inference rules in a symbolic logic is, in these cases, replaced by the problem of drawing particular diagrams correctly.^{[5]} For instance, an Euler Circles diagram ventures to capture relationships between sets using topological relationships between plane regions in such a way that that it depicts all the possible ways that a certain collection of settheoretic statements could be true. This has two important consequences: (1) if a certain diagram cannot be drawn then the described situation must be impossible (termed “selfconsistency”), and (2) if a certain relationship between diagram objects must be drawn, then the corresponding relation can be inferred as logically vaild. (See the numerous examples in Section 2.) This phenomenon is often termed a “freeride” [Barwise & Shimojima 1995]. This style of diagrammatic reasoning is thus dependent on a particular representational use of diagrams — that they represent classes of models. If a particular class of models cannot be represented by a diagrammatic system, then those cases will not be taken into account in inferences using the system, and incorrect inferences might be drawn. This fact makes the representational adequacy of diagrammatic systems, restricted by their spatial nature, of paramount importance, as we shall now explore.
The representational use of the spatial relations in the plane constrains diagrammatic representation, and therefore reasoning with diagrams, in certain important ways. In particular, there are topological and geometrical (let us lump them together as “spatial") properties of diagrammatic objects and relations which limit the expressive power of diagrammatic systems. For instance, in graph theory it is known that some simple structures cannot be drawn in the plane. For example, the graph K_{5} is the graph consisting of 5 nodes, each joined to the other by an arc. This graph is nonplanar, meaning that it cannot be drawn without at least two of the arcs crossing. This is just the sort of constraint on possible diagrams that limits the expressive power of diagrammatic systems. Now, since diagrammatic reasoning can occur by enumeration of all possible models of a situation, this representational inadequacy (a type of incompleteness) renders many diagrammatic systems incorrect if they are used for logical reasoning (e.g., see the critique of [Englebretsen 1992] in [Lemon & Pratt 1998]).
Perhaps the most simple example of this is due to Lemon and Pratt^{[6]} (see e.g., [1997]). Consider Euler Circles — where convex regions of the plane represent sets, and overlap of the regions represents nonempty intersection of the corresponding sets. A result of convex topology known as Helly's Theorem states (for the 2 dimensional case) that if every triple of 4 convex regions has a nonempty intersection then all four regions must have a nonempty intersection.
To understand the ramifications of this, consider the following problem:
Example 4. Using Euler Circles, represent the following premises:Note that, in terms of settheory, only trivial consequences follow from these premises. However, an Euler diagram of the premises, such as Figure 5, leads to the incorrect conclusion that A B C D (due to the quadruple overlap region in the centre of the diagram):A B CB C D
C D A
Figure 5: An Euler's Circles representation exhibiting Helly's Theorem
In other words, a user of Euler Circles is forced^{[7]} to represent a relationship between the sets which is not logically necessary. This means both that there are logically possible situations which the system cannot represent, and that a user would make incorrect inferences if they relied on the system for reasoning. More generally, this type of result can be generated for many different types of diagrammatic system, depending on the particular spatial relations and objects which they use in representation — a research programme which is ongoing.
For example, using nonconvex regions (e.g., “blobs” instead of circles) leads to a similar problem, only that nonplanar graphs are involved instead of Helly's Theorem. A similar result concerns linear diagrams for syllogisms [Englebretsen 1992], where lines are used to represent sets, points represent individuals, pointline intersection represents setmembership, and intersection of lines represents setintersection. Again, planarity constraints restrict the expressive power of the system and lead to incorrect inferences.
Atsushi Shimojima's “constraint hypothesis” perhaps best sums all this up:
Representations are objects in the world, and as such they obey certain structural constraints that govern their possible formation. The variance in inferential potential of different modes of representation is largely attributable to different ways in which these structural constraints on representations match with the constraints on targets of representation ([Shimojima 1996a, 1999]).
As discussed above, much of the interest in diagrams has been generated by the claim that they are somehow more “effective” than traditional logical representations for certain types of task. Certainly, for example, a map is a greater aid to navigation than a verbal description of a landscape. However, while there are certainly psychological advantages to be gained through the use of diagrams, they are (as in the case of Euler Circles) often ineffective as representations of abstract objects and relationships. Once a purely intuitive notion, nonpsychological claims about “efficacy” of diagrammatic systems can be examined in terms of standard formal properties of languages [Lemon et al. 1999]. In particular, many diagrammatic systems are selfconsistent, incorrect, and incomplete, and complexity of inference with the diagrams is NPhard. By way of contrast, most sentential logics, while able to express inconsistencies, are complete and correct^{[8]} .
On the other hand, not being able to represent contradictions could provide us with interesting insights about the nature of diagrammatic representation. If a central goal of a language is to represent the world or a state of affairs, then representing contradictions or tautologies is called into question. Neither contradictions nor tautologies are part of the world. How can we draw a picture, or take a picture, of the contradiction that “it is raining and it is not raining”? How about the picture of the disjunctive information “it is either raining or not raining”? Now, we seem to be much closer to Wittgenstein's classic picture theory of language [Wittgenstein 1921].
On a more practical theme, AI researchers, one of whose main concerns is the heuristic power of representation systems in addition to their expressive power, have been debating about different forms of representation for decades [Sloman 1971, 1985, 1995]. Hence, they have welcomed discussions of the distinct role of visual reasoning and have recently hosted interdisciplinary symposiums on diagrammatic reasoning at AI conferences.^{[9]} At the same time, realizing that human beings adopt different representation forms depending on the kinds of problems they face, some AI researchers and design theorists have practiced domainspecific approaches to bringing in problemtailored representation forms.^{[10]}
For instance, Harel [1988] invented higraphs to represent system specifications in computer science. This idea has been taken up in industrial applications (e.g., UML [Booch et al. 1998]). BarkerPlummer & Bailin 1997 present a case study in developing computers which can carry out the kind of analogical reasoning that humans perform when proving certain mathematical theorems. More recently, an interesting result has been presented by Mateja Jamnik of Alan Bundy's Mathematical Reasoning Group at Edinburgh [Jamnik 2001]. Jamnik shows how a semiautomatic formal proof system can perform some of the perceptual inferences which humans find so natural. For instance, that the sum of the first n odd natural numbers is n squared is easily seen by decomposing an n x n grid into “ells” [Jamnik et al. 1999].
It should also be mentioned that scientists such as chemists and physicists also use diagrams in order to perform certain computations. Feynman diagrams, for example, are used to perform calculations in subatomic physics. In Knot Theory (which has applications in physics [Kauffman 1991]) the three Reidemeister Moves are diagrammatic operations which make up a complete calculus for proving knots equivalent.
Do our mental representations have diagramlike or picturelike entities as components? This question has a long history both in philosophy and in psychology, independently of each other. More recently, however, some philosophers have participated in this “imagery debate”, one of the most timehonored controversies in psychology, and some cognitive psychologists find certain epistemological theories in philosophy useful to support their views on the issue.
The nature of mental representation has been one of the perennial topics in philosophy, and we can easily trace back philosophical discussions on images and mental representation to ancient times.^{[11]} The writings of Hobbes, Locke, Berkeley, and Hume concern themselves in large part with mental discourse, the meaning of words, mental images, particular ideas, abstract ideas, impressions, and so on. Descartes' wellknown distinction between imagining and conceiving something has generated much discussion about the unique role of visual images in mental representations. The development of cognitive science in the 20th century naturally has brought certain group of philosophers and psychologists closer and we find a number of authors whose works easily belong to both disciplines [Block 1983, Dennett 1981, Fodor 1981].
Imagery based on introspection was the main focus in the early development of psychology until the behavioristic approach became predominant in the discipline. During the era of behaviorism, anything related to mental inspection, including images, was excluded from any serious research agenda. Finally when the topic of mental images made a comeback in psychology in 1960s, researchers adopted a more humble agenda for mental imagery than before: Not all mental representations involve imagery, and imagery is one of many ways of manipulating information in the mind. Also, thanks to the influence of behaviorism, it is acknowledged that introspection is not enough to explore imagery, but a claim about mental imagery needs to be confirmable by experiments in order to show that we successfully externalize mental events. That is, if what a certain mental introspection tells us is genuine, then there would be observable external consequences of that mental state.
Thus the contemporary imagery debate among cognitive scientists is about the claim that picturelike images exist as mental representations and about how we interpret certain experiments.^{[12]}
Kosslyn [1980, 1994] and other pictorialists [Shepard & Metzler 1971] present experimental data to support their position that some of our mental images are more like pictures than a linear form of language (for example, natural languages or artificial symbolic languages) in some important aspects, even though not all visual mental images and pictures are of exactly the same kind. By contrast, Pylyshyn [1981] and other descriptionalists [Dennett 1981] raise questions about the picturelike status of mental images and argue that mental images are formed out of structured descriptions. To them, mental images represent in the manner of language rather than pictures and, hence, there are no picturelike visual mental images.
Both sides of the debate sometimes used a philosophical theory as a supporting factor. For example, pictorialists in the imagery debate found the modern sensedatum theory in philosophy quite close to their point of view. By the same token, the critics of the sensedatum theory argued that the mistaken pictorial view of mental images arises mainly from our confusion about ordinary language and claimed that mental images are epiphenomena.
Without being heavily involved in the imagery debate, some researchers have focused on a distinct role that diagrams or pictures — as opposed to traditional sentential forms — play in our cognitive activities. Based on the conjectures that humans adopt diagrammatic or spatial internal mental representations in their reasoning about concrete or abstract situations (see [Howell 1976, Sober 1976]), some cognitive scientists have concentrated on the functions of images or diagrams in our various cognitive activities, for example, memory, imagination, perception, navigation, inference, problemsolving, and so on. Here, the distinct nature of “visual information,” which is obtained either through internal mental images or through externally drawn diagrams, has become a major topic of research. Even though most of these works assume that there are mental images (that is, they accept the pictorialists' claim), strictly speaking they do not have to commit themselves to the view that these images exist as basic units in our cognition. Descriptionalists do not have to discard discussions of the functions of images, but only need to add that these images are not primitive units stored in our memory, but formed out of structured descriptions more like the sentences of a language. (See [Pylyshyn 1981].)
A search for the distinct role of diagrams has led researchers to explore the differences among different forms of external or internal representations, and mainly between diagrammatic and sentential representations. Many important results have been produced in cognitive science. Starting from Larkin and Simon's classic case study [1987] to illustrate a difference between informational and computational equivalence among representation systems, Lindsay's work locates where this computational difference lies, which he calls a ‘nondeductive’ method. As briefly pointed out above, this inference process is called ‘free ride’ by Barwise and Shimojiima [1995], i.e., the kind of an inference in which the conclusion seems to be read off automatically from the representation of premises. In Gurr, Lee, and Stenning [1998] and Stenning and Lemon [2001], there is an explanation of the uniqueness of diagrammatic inference in terms of a degree of ‘directness’ of interpretation, and it is argued that this property is relative, and hence that “some rides are cheaper than others”. Having the role of graphs in mind, Wang and Lee ([1993]) present a formal framework as a guideline for correct visual languages. At this point, we are very close to applied aspects of research in multimodal reasoning — design theory and AI research — by providing these disciplines with computational support for visual reasoning.
Related to the issue of imagistic mental representation is the examination of the semantics of various diagrammatic systems and what they can teach us about the nature of languages in general (e.g., Goodman [1968]). For instance, Robert Cummins [1996], amongst others, argues that too little attention has been given to diagrammatic representations and that focus on a notion of “structural representation” more akin to diagrammatic representation can help to explicate the nature of representation itself. We believe that the considerations presented above give us some empirical handle on this type of claim at least — depending on the imagistic objects and relations used, patterns of incorrect inference should be predictable and detectable. An important, if littleknown, article on this theme is [Malinas 1991]. Here Malinas explores the concepts of pictorial representation and “truth in” a picture via the notion of resemblance, and considers various semantic puzzles about pictorial representation. He develops Peacocke's “Central Thesis” of depiction ([Peacocke 1987]), where experienced similarities between properties of pictorial objects and their referents in the visual field give rise to the relation of depiction. He goes on to provide a formal semantics for pictures which is “analogous to a semantics for an ideal language”.
SunJoo Shin SunJoo.Shin@yale.edu  Oliver Lemon olemon@inf.ed.ac.uk 
A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z