Category Archives: Interdisciplinary research

Research that crosses disciplines and does not fit easily into one category.

Human-centered AI Systems

When considering ethical AI,  I find it useful to distinguish between two types of AI systems, each of which have different ethical and dependability considerations.

Type 1: “Nature inspired” AI systems: their purpose is to help understand and replicate key features of natural intelligence. They may be simulations or experimental hardware robots. Examples include neuroscience models, swarm intelligence, artificial societies and cognitive architectures. They are not deployed in environments where humans depend on them. Although these models would mostly be developed for research purposes, it is conceivable that they might be applied for a purpose (e.g. games, virtual characters, exploring Mars).

Type 2: “Human-centred” AI systems: they are developed for a particular purpose within a human environment (e.g. an organisation or a home).  Examples include medical decision support systems and home robots. These systems need to be dependable, particularly if they involve autonomous actions. They may contain biologically inspired models originally developed as Type 1 research models (e.g. cognitive architectures, neural network learning), but only if such models can be applied effectively to satisfy the system requirements.

Importance of requirements
The two categories above are approximate and not mutually exclusive. For example, a Type 1 system may involve hardware robots with experimental human interaction (e.g. humans teaching a robot the names of objects and how to manipulate them). In such cases, safety may become important, but to a lesser extent than for most Type 2 systems. Such a robot could misidentify objects or misinterpret human communication without serious consequences. The main difference between the two classes is that the requirements of a Type 2 system are specified by humans who depend on it, while the main requirement for a Type 1 system is to survive in a challenging environment or to be successful at solving problems that biological systems can solve. Both these requirements can be combined in Type 2 systems (but the human-specified requirements would take precedence).

Knowledge and communication
For Type 1 systems, any knowledge that the system acquires would be relevant for its goals but not necessarily relevant for humans.  For example, an agent-based model might be used to study the evolution of communication in a simulated society. The evolving language constructs would be used to label entities in the simulated world which are relevant for the agents, but the labels (and possibly the labelled entities) might not be meaningful for humans. Similarly, applied systems such as a Mars explorer may develop concepts which humans don’t use and need not be explained (unless they become relevant for human exploration). These AI systems would acquire their knowledge autonomously by interacting with their environment (although some innate “scaffolding” may be needed).

For Type 2, the AI system’s knowledge needs to connect with human concepts and values. Knowledge acquisition using machine learning is a possibility, but there are debates on its limitations. A good discussion is in “Rebooting AI”  [Marcus and Davis 2019]. Even if machine learning of human values were possible technically, it might not be desirable. See for example, the discussions in [Dignum 2019] where “bottom-up” learning of values is contrasted with “top-down” specification (hybrids are possible).  I think there is an important role for human participation and expertise in the knowledge acquisition process, and it makes sense for some knowledge to be hand-crafted (in addition to learning). In future posts, I plan to explore what this means for system design in practice.

  • [Dignum 2019] Dignum, V. (2019) Responsible Artificial Intelligence. Springer.
  • [Marcus and Davis 2019] Marcus, G and Davis, E.  (2019).  Rebooting AI: Building Artificial Intelligence We Can Trust. Pantheon Books, USA.

What is a Cognitive Architecture?

Some of my research involves the computational modelling of cognition and how it interacts with emotion. Computational modelling is useful for the study of human or animal cognition, as well as for the building of artificial cognitive systems (e.g. robots).  The cognitive process being modelled may be understood as an autonomous system which senses information from its environment and uses this information to determine its next action. Such an autonomous system is often called an “agent” [Russel and Norvig 2010].  A cognitive architecture is a specification of the internal structure of a cognitive agent, defining the components of cognition and their interactions. The concept of “architecture” is important because it integrates the various functions of cognition into a coherent system. Such integration is necessary for building complete autonomous agents and for the study of interactions between different components of natural cognition, such as reasoning and motivation.

Multiple Levels
Architectures can be defined at different levels of detail. For example, [Marr 1982] defines three levels which can be applied to cognitive architecture as follows:
Level 1: “Computational theory”: this specifies the functions of cognition – what components are involved and what are their inputs and outputs?
Level 2:. “Representation and algorithm”: this specifies how each component accepts its input and generate its output. For example, representations may include symbolic logic or neural nets; algorithms may include inference algorithms (for logical deductions) or learning algorithms.
Level 3: “Implementation”: this specifies the hardware, along with any supporting software and configurations (e.g. simulation software, physical robot or IT infrastructure).

At level 1, the architecture specifies the components and their interfaces. For example, a perception component takes raw sense data as input and identifies objects in a scene; a decision component generates an action depending on objects identified by the perception component. Level 2 fills in the detail of how these components work. For example, the perception component might generate a logic-based representation of the raw data that it has sensed, while the decision component uses logic-based planning to generate actions. Level 3 provides an executable instantiation of the architecture. An instantiation may be a physical robot, a software product or prototype, or a model of an agent/robot which can be run as a simulation on a particular platform.

Environment and requirements
When designing an architecture, the environment of the agent needs to be considered. This defines the situations and events that the agent encounters. It is also important to define requirements that the agent must satisfy in a given environment. These may be capabilities (e.g. to detect the novelty of an unforeseen situation or to act on behalf of human values sufficiently accurately so that humans can delegate tasks to the agent in the given environment). If a natural system is being modelled (e.g. an animal), the requirements may simply be survival in the given environment. Assumptions made about the environment help to constrain the requirements.

Architecture examples
Example architectures that are particularly relevant to my research include H-CogAff [Sloman et al. 2005] and MAMID [Hudlicka 2007]. Both are modelling human cognition. H-CogAff emphasises the difference between fast instinctive reactions and slower reasoning. MAMID focuses on emotion generation and its effects on cognition.  Architectures need not necessarily be executable (i.e. defined at levels 1 to 3). For example, H-CogAff is not a complete architecture that can be translated into an executable instance, but it is a useful guideline.

Broad-and-shallow architectures
Executable architectures can be developed using iterative stepwise refinement, beginning with simple components and gradually increasing their complexity. The complexity of the environment can also be gradually increased. To experiment with ideas quickly, it is important to use a rapid-prototyping methodology. This allows possibilities to be explored and unforeseen difficulties to be discovered early. To enable rapid-prototyping, an architecture should be made executable as early as possible in the development process. A useful approach is to start with a “broad and shallow” architecture [Bates et al. 1991].  This kind of architecture is mostly defined at level 1, with artificially simplified levels 2 and 3. For example, at level 2, the perception component may be populated temporarily by a simple data query method (does this object exist in the data?) and the decision component might include simplified “if-then” rules. For level 3, a simulation platform may be used which is suitable for rapid-prototyping.

In later posts, I will discuss how this methodology fits in with AI research more generally and ethical AI systems in particular.


  • [Russell and Norvig 2010] Russell, S. J., Norvig, P., & Davis, E. (2010). Artificial intelligence: A modern approach.
  • [Marr 1982] Marr, D. (1982), Vision: A Computational Approach, San Francisco, Freeman & Co. Full text:]_Vision_A_Computational_Investigation.pdf
  • [Bates et al. 1991] Bates, J., Loyall, A. B., & Reilly, W. S. (1991). Broad agents. Proceedings AAAI Spring Symposium on Integrated Intelligent Architectures. Stanford, CA: Reprinted in Sigart Bulletin, 2(4), Aug. 1991, pp. 38-40.)
  • [Sloman et al. 2005] Sloman, A., Chrisley, R., Scheutz, M. (2005). The Architectural Basis of Affective States and Processes. In: Fellous, J.-M., Arbib, M.A. (eds.) Who Needs Emotions? New York: Oxford University Press.
    Full text:
  • [Hudlicka 2007] Hudlicka, E.(2007): Reasons for emotions: Modeling emotions in integrated cognitive systems. In W. Gray (Ed.), Integrated Models of Cognitive Systems, 137. New York:Oxford University Press.