Publications and software

Hello I am Peter Cheng.

This is a brief overview of the paper ‘A sketch of a theory and modelling notation for elucidating the structure of representations’ which is presented at the Diagrams Conference 2020.

I hope this will tempt you to delve into the paper.

What is a structural theory of symbolic graphical representations? Well it is an explanation of the nature of the fundamental of the cognitive components of all representations. And how these components are put together, composed into structures that capture meaning.

The modelling notation is a tool or a language to help elucidate specific structures of graphical displays in particular domains. The theory was inspired by an analogy to molecular theory in Chemistry. What it does is to try and identify the basic elements of all representations and how they are assembled into larger molecular structures of the whole representations.

There aren’t any theories that I know of in the literature that systematically address the structure of representations in detail, and so this is the first attempt of such a theory, which is why I’ve put ’sketch’ in the title.

What should such a cognitive theory of symbolic graphical representations cover? I’ll answer these in two ways. First, the scope should span the full diversity of representations. Here are some examples which I cover in the paper, which range from simple icons through to mathematical equations and onto simple and complex charts. The Monster Graph on the bottom right is from engineering thermodynamics. The second way to answer the question ‘what should the theory cover?’ is by thinking of the kinds of questions that it should address. So, here are some examples. How are these representations shown here different? But also, they are fundamentally similar because their function is to serve as representations. So, in which ways are they similar? What’s the fundamental set of cognitive entities or schemas that constitute all representations? Can the theory be used to predict whether a representation would be easy or difficult to use? And here is a real challenge. Can a theory serve as some kind of basis for a system that would systematically allows us to design effective novel graphical representations? I do not think there is an existing theory that has a potential to be able to address all of these questions.

The theory proposes that just three, three basic kinds of elements are used to build any representations. These elements are cognitive schemas and the theory defines the structure in terms of slots and fillers. Each schema specifies how concepts are encoded in graphical structures at different levels of granularity to serve different kinds of representational functions. Here is an example of a structural diagram in the modelling notation for a bar chart.

The paper demonstrates the utility of the theory by applying it to complex representations. The analysis of the algebraic equation shows its hidden complexity. Although the structure of the Monster Thermodynamic Graph is complex, the analysis in terms of the theory and notation actually shows that it is simple in some fundamental ways compared with the algebraic equation.

In summary, the paper covers these things. The requirements of a structural theory of representations. The fundamental components of representations. The modelling notation that says how these components should be put together. Analysis guidelines for interpreting the actual structure of representations. Lots of examples of diverse structure of representations that suggest the validity of the theory. And a brief discussion of the potential uses of the theory and notation.

I hope you are intrigued enough to read the paper and would find it interesting.

Thank you.

Tactile Protocol Analysis: Coding Scheme Demo.

The following codes are shown for behaviours when reading tactile graphics: Scan, Comb, Skim, SlideTo, JumpTo, Scan-Within, Trace, Glance, Span,fix, Brush, Tap, Hover, Park, Break.

Code: Scan.
Definition: Move across a distinct sequence of objects in order; e.g., row, column.
Action: Between objects
Contact: There is contact with the surface.
Pause: (May) Pause at intermediate objects.
Movement: Systematic.

Code; Comb.
Definition: Set of successive actions occurring in a meaningful area; e.g., 3x2 adjacent objects; e.g., following axis. Systematic actions performed at some spatially organised objects. May consists of multiple scans.
Action: Between objects.
Contact: There is contact with surface.
Pause: (May) Pause at intermediate objects.
Movement: Systematic.

Code: Skim Definition:
Movement in contact with surface without single definite end object or location. Not following a particular sequence of objects. It may involve palms.
Action: Between objects.
Contact: There is contact with surface.
Pause: Do not pause at intermediate objects.
Movement: Arbitrary, in multiple directions.

Code: SlideTo.
Definition: Move to digit(s) between a start and end object/location.
Action: Between objects.
Contact: There is contact with the surface.
Pause: Do not pause at intermediate objects.
Movement: Systematic.

Code: JumpTo
Definition: Move of digit(s) between specific start and end object/location, breaking contact with surface.
Action: Between objects.
Contact: Mostly, no contact with surface.
Pause: Do not pause at intermediate objects.
Movement: Systematic.

Code: Scan-within.
Definition: One movement on any direction while touching the surface. Typically observed within a fraction of a second.
Action: Within an object.
Contact: There is contact with surface.
Pause: Do not pause.
Movement: Arbitrary, in any direction.

Code: Trace.
Definition: Touch on part or whole of a line or perimeter of object. Single direction of motion, no backtrack.
Action: Within an object.
Contact: There is always contact with surface.
Pause: (May) Pause at intermediate objects.
Movement: Systematic.

Code: Glance.
Definition: Touch a point of an object without stopping, but perhaps slowing.
Action: Within an object.
Contact: There is always contact with surface.
Pause: Do not pause.
Movement: Arbitrary, in any direction.

Code: Span.
Definition: Stretch digits of one hand between original and new location keeping some contact with original location. Typically happens within milliseconds.
Action: Within an object or across objects.
Contact: There is always contact with surface.
Pause: Do not pause.
Movement: Systematic.

Code: Fix.
Definition: Hold stationary touching surface. The surface could be touched before, after or both for this action.
Action: Within an object or in blank.
Contact: There is always contact with the surface.
Pause: Stays in one location.
Movement: None.

Code: Brush.
Definition: Back and forth (2 or more traces/glances in opposite directions) touches continuously on pat or whole of a line or object. Varying direction of motion.
Action: Within an object.
Contact: There is always contact with surface.
Pause: Do not pause.
Movement: Systematic, it may follow a contour.

Code: Tap.
Definition: Movement point touch(es), but no horizontal movement. Touches are in the same location.
Action: Over/in an object.
Contact: Partial contact with surface.
Pause: N/A
Movement: N/A

Code: Hover.
Definition: A lift off and return to the same object, with no intervening translation of finger.
Action: Over an object.
Contact: No contact with surface.
Pause: N/A
Movement: Stationary.

Code: Park
Definition: Hand stationary directly above reading surface, not on surface.
Action: over display.
Contact: No contact with surface.
Pause: Do not pause.
Movement: Stationary.

Code: Break.
Definition: Hand moves away from the surface.
Action: Away from display.
Contact: No contact with surface.
Pause: N/A
Movement: N/A

Cognitive Science of Tactile Graphics.

https://www.sussex.ac.uk/rmphillips/research/tactilegraph

Project Funded By RM Phillips Research Network. Sussex University.

Project Leaders: Prof. Peter Cheng, Dr. Frances Aldrich.

Research Fellows: Dr. Ronald Grau, Dr. Grecia Garcia

Copyright, 2017 Representational Systems Lab. Sussex University.