Tag Archives: software engineering

Human-centered AI Systems

When considering ethical AI,  I find it useful to distinguish between two types of AI systems, each of which have different ethical and dependability considerations.

Type 1: “Nature inspired” AI systems: their purpose is to help understand and replicate key features of natural intelligence. They may be simulations or experimental hardware robots. Examples include neuroscience models, swarm intelligence, artificial societies and cognitive architectures. They are not deployed in environments where humans depend on them. Although these models would mostly be developed for research purposes, it is conceivable that they might be applied for a purpose (e.g. games, virtual characters, exploring Mars).

Type 2: “Human-centred” AI systems: they are developed for a particular purpose within a human environment (e.g. an organisation or a home).  Examples include medical decision support systems and home robots. These systems need to be dependable, particularly if they involve autonomous actions. They may contain biologically inspired models originally developed as Type 1 research models (e.g. cognitive architectures, neural network learning), but only if such models can be applied effectively to satisfy the system requirements.

Importance of requirements
The two categories above are approximate and not mutually exclusive. For example, a Type 1 system may involve hardware robots with experimental human interaction (e.g. humans teaching a robot the names of objects and how to manipulate them). In such cases, safety may become important, but to a lesser extent than for most Type 2 systems. Such a robot could misidentify objects or misinterpret human communication without serious consequences. The main difference between the two classes is that the requirements of a Type 2 system are specified by humans who depend on it, while the main requirement for a Type 1 system is to survive in a challenging environment or to be successful at solving problems that biological systems can solve. Both these requirements can be combined in Type 2 systems (but the human-specified requirements would take precedence).

Knowledge and communication
For Type 1 systems, any knowledge that the system acquires would be relevant for its goals but not necessarily relevant for humans.  For example, an agent-based model might be used to study the evolution of communication in a simulated society. The evolving language constructs would be used to label entities in the simulated world which are relevant for the agents, but the labels (and possibly the labelled entities) might not be meaningful for humans. Similarly, applied systems such as a Mars explorer may develop concepts which humans don’t use and need not be explained (unless they become relevant for human exploration). These AI systems would acquire their knowledge autonomously by interacting with their environment (although some innate “scaffolding” may be needed).

For Type 2, the AI system’s knowledge needs to connect with human concepts and values. Knowledge acquisition using machine learning is a possibility, but there are debates on its limitations. A good discussion is in “Rebooting AI”  [Marcus and Davis 2019]. Even if machine learning of human values were possible technically, it might not be desirable. See for example, the discussions in [Dignum 2019] where “bottom-up” learning of values is contrasted with “top-down” specification (hybrids are possible).  I think there is an important role for human participation and expertise in the knowledge acquisition process, and it makes sense for some knowledge to be hand-crafted (in addition to learning). In future posts, I plan to explore what this means for system design in practice.

  • [Dignum 2019] Dignum, V. (2019) Responsible Artificial Intelligence. Springer.
  • [Marcus and Davis 2019] Marcus, G and Davis, E.  (2019).  Rebooting AI: Building Artificial Intelligence We Can Trust. Pantheon Books, USA.

What is a Cognitive Architecture?

Some of my research involves the computational modelling of cognition and how it interacts with emotion. Computational modelling is useful for the study of human or animal cognition, as well as for the building of artificial cognitive systems (e.g. robots).  The cognitive process being modelled may be understood as an autonomous system which senses information from its environment and uses this information to determine its next action. Such an autonomous system is often called an “agent” [Russel and Norvig 2010].  A cognitive architecture is a specification of the internal structure of a cognitive agent, defining the components of cognition and their interactions. The concept of “architecture” is important because it integrates the various functions of cognition into a coherent system. Such integration is necessary for building complete autonomous agents and for the study of interactions between different components of natural cognition, such as reasoning and motivation.

Multiple Levels
Architectures can be defined at different levels of detail. For example, [Marr 1982] defines three levels which can be applied to cognitive architecture as follows:
Level 1: “Computational theory”: this specifies the functions of cognition – what components are involved and what are their inputs and outputs?
Level 2:. “Representation and algorithm”: this specifies how each component accepts its input and generate its output. For example, representations may include symbolic logic or neural nets; algorithms may include inference algorithms (for logical deductions) or learning algorithms.
Level 3: “Implementation”: this specifies the hardware, along with any supporting software and configurations (e.g. simulation software, physical robot or IT infrastructure).

At level 1, the architecture specifies the components and their interfaces. For example, a perception component takes raw sense data as input and identifies objects in a scene; a decision component generates an action depending on objects identified by the perception component. Level 2 fills in the detail of how these components work. For example, the perception component might generate a logic-based representation of the raw data that it has sensed, while the decision component uses logic-based planning to generate actions. Level 3 provides an executable instantiation of the architecture. An instantiation may be a physical robot, a software product or prototype, or a model of an agent/robot which can be run as a simulation on a particular platform.

Environment and requirements
When designing an architecture, the environment of the agent needs to be considered. This defines the situations and events that the agent encounters. It is also important to define requirements that the agent must satisfy in a given environment. These may be capabilities (e.g. to detect the novelty of an unforeseen situation or to act on behalf of human values sufficiently accurately so that humans can delegate tasks to the agent in the given environment). If a natural system is being modelled (e.g. an animal), the requirements may simply be survival in the given environment. Assumptions made about the environment help to constrain the requirements.

Architecture examples
Example architectures that are particularly relevant to my research include H-CogAff [Sloman et al. 2005] and MAMID [Hudlicka 2007]. Both are modelling human cognition. H-CogAff emphasises the difference between fast instinctive reactions and slower reasoning. MAMID focuses on emotion generation and its effects on cognition.  Architectures need not necessarily be executable (i.e. defined at levels 1 to 3). For example, H-CogAff is not a complete architecture that can be translated into an executable instance, but it is a useful guideline.

Broad-and-shallow architectures
Executable architectures can be developed using iterative stepwise refinement, beginning with simple components and gradually increasing their complexity. The complexity of the environment can also be gradually increased. To experiment with ideas quickly, it is important to use a rapid-prototyping methodology. This allows possibilities to be explored and unforeseen difficulties to be discovered early. To enable rapid-prototyping, an architecture should be made executable as early as possible in the development process. A useful approach is to start with a “broad and shallow” architecture [Bates et al. 1991].  This kind of architecture is mostly defined at level 1, with artificially simplified levels 2 and 3. For example, at level 2, the perception component may be populated temporarily by a simple data query method (does this object exist in the data?) and the decision component might include simplified “if-then” rules. For level 3, a simulation platform may be used which is suitable for rapid-prototyping.

In later posts, I will discuss how this methodology fits in with AI research more generally and ethical AI systems in particular.


  • [Russell and Norvig 2010] Russell, S. J., Norvig, P., & Davis, E. (2010). Artificial intelligence: A modern approach.
  • [Marr 1982] Marr, D. (1982), Vision: A Computational Approach, San Francisco, Freeman & Co. Full text: http://s-f-walker.org.uk/pubsebooks/epubs/Marr]_Vision_A_Computational_Investigation.pdf
  • [Bates et al. 1991] Bates, J., Loyall, A. B., & Reilly, W. S. (1991). Broad agents. Proceedings AAAI Spring Symposium on Integrated Intelligent Architectures. Stanford, CA: Reprinted in Sigart Bulletin, 2(4), Aug. 1991, pp. 38-40.)
  • [Sloman et al. 2005] Sloman, A., Chrisley, R., Scheutz, M. (2005). The Architectural Basis of Affective States and Processes. In: Fellous, J.-M., Arbib, M.A. (eds.) Who Needs Emotions? New York: Oxford University Press.
    Full text: http://www.sdela.dds.nl/entityresearch/sloman-chrisley-scheutz-emotions.pdf
  • [Hudlicka 2007] Hudlicka, E.(2007): Reasons for emotions: Modeling emotions in integrated cognitive systems. In W. Gray (Ed.), Integrated Models of Cognitive Systems, 137. New York:Oxford University Press.

Integrity in collaborative IT systems: Part 2 – the need for rich test environments

In Part 1, I argued that dependability as a concept might be applied to organisations as well as to technical systems. In this post I will argue that both the organisational and technical levels should be modelled together as an interconnected system, and that test environments for dependability should include the simulation of organisational problems as well as technical problems.

Socio-technical stack
Higher level organisational requirements cannot be considered in isolation from the underlying IT requirements. Organisational and IT system problems can interact in complex ways and such problems are common in real-world organisations. Therefore, these different levels need to be considered together. Such a multi-level system can be viewed as a socio-technical stack [Baxter & Sommerville 2011].

The different levels of requirements can be listed as follows:

  1. Specific organisational functionality requirements (e.g. medical workflows)
  2. Organisational dependability requirements (e.g. avoiding error)
  3. Specific IT requirements for the organisation (resources, networks etc.)
  4. IT dependability requirements (availability, security etc.)

Dependability requirements (2 and 4) may be more generic than 1 and 3. For example, all organisations will want to reduce error, but they may have different measures of what is acceptable. Requirements 3 and 4 can usually be satisfied by off-the-shelf components (but would need to be configured).

We assume that the software to satisfy the first set of requirements (1) has multiple users with different services. Such software is often called “enterprise application software”. In a health care system, users can be patients, clinicians or administrators. They access their own services in the system and they have specific actions available to them at particular stages in their workflow. For example, a patient could review their details or access records following a consultation. A clinician could request a test or respond to a symptom update from a patient.

Need for a test environment with simulation
To improve organisational resilience and dependability, it is important to develop new methods for detection and correction of organisational problems. To test these problem detection and recovery methods, it is useful to run simulated scenarios where human mistakes and IT failures can occur together. “Simulations” might involve people participating (as in a kind of role-playing game) or simulated computational agents [Macal 2016].

Examples of failure that might be simulated:

  • mistakes (e.g. choosing the wrong test)
  • administration failure: patient receives no response to a request (which should have a time limit).
  • software failure: e.g. data interoperability issues.
  • malware
  • hardware failure

A test environment needs to be coupled with the iterative development of the system being tested. This would involve the development of increasingly complex problem-detection software in parallel with increasingly challenging scenarios. For example, the first version might involve simple errors that are easy to detect. Subsequent stages might involve increasingly more detailed work scenarios with more complex errors or failures. The more advanced stages might also involve real users in different roles (e.g. nursing students, medical students) and include time pressure.

Importance of agile and participatory design
In addition to developing safe systems, changing them safely is also important. So the development and test methodology needs to include change management. Agile software engineering is particularly important here, along with participatory design (co-design) methods. Ideally the system would be co-designed iteratively by the different users as they become aware of error-prone situations (such as cognitive overload) while participating in the evaluations. Design needs be informed by cognitive science as well as computer science.

In later posts, I plan to talk about the role of AI and decision support in organisational dependability.


Integrity in collaborative IT systems: Part 1 – the concept of dependability

Recently I’ve been looking at collaborative decision-making in mental health, with the aim of identifying the technology requirements to support shared decision-making. Details of this project are here). One conclusion is that the underlying IT infrastructure needs to be considered, and in particular its reliability.

In general, a collaborative IT system can be understood as a distributed system with a particular purpose, where users with different roles collaborate to achieve a common goal. Examples include university research collaboration, public transport and e-government. In the example of health IT, a medical practice might have an IT system where a patient makes an appointment, medical records are inspected and updated, treatment decisions are made and recorded, and the patient may be referred to a specialist.

IT resilience and dependability
The resilience of an IT system is its capability to satisfy service requirements if some of its components fail or are changed. If parts of the system fail due to faults, design errors or cyber-attack, the system continues to deliver the required services. Similarly, if a software update is made, the system services should not be adversely affected. Resilience is an important aspect of dependability, which is defined precisely in terms of availability, reliability, safety, security and maintainability [Avizienis et al. 2004]. Importantly, dependability is not just about resilience, but also about trust and integrity.

IT dependability is usually understood on a technical level (the network or the software) and does not consider the design of the organisation (for example, if an error occurs due to lack of training).

Organisational resilience and dependability
Just as an IT system can be resilient on a technical level, an organisation (such as a health provider) can also be resilient and dependable in meeting high-level organisational requirements. Organisational requirements are defined in terms of an organisation, and are independent of IT. For example, they may be defined in terms of business processes or workflows. I think the idea of dependability requirements for an organisation is also useful and these may be specified separately. In healthcare, they might include the following:

  • implementation – ensure that agreed decisions are actually carried out.
  • avoidance of error – e.g. avoid excessive workloads.
  • timeliness (e.g. for cancer diagnosis)
  • transparency – e.g. is there an audit trail of critical decisions and actions?
  • accountability – e.g. is it possible to challenge decisions?

Technology can help to ensure that these dependability requirements are satisfied. For example, excessive workload may be detectable by automated monitoring (e.g. one person doing too many tasks) in the same way that technical faults or security violations can be detected.

In Part 2, I will discuss the need for a test and simulation environment.

[Avizienis et al. 2004] Avizienis A, Laprie J-C, Randell B, and Landwehr C, “Basic concepts and taxonomy of dependable and secure computing,” IEEE Transactions on Dependable and Secure Computing, vol. 1, no. 1, pp. 11-33, Jan.-March 2004.

Smart Assistance for Lifestyle Decisions

Smart assistance technology helps users to achieve goals using informative messages such as recommendations or reminders. To be “smart” the messages need to be sensitive to the user’s changing context, including their current knowledge and perceptions, their mood and environment.

Right now, I’m starting to explore the feasibility of a “smart assistance” app to support users with everyday decisions, for example with food, shopping, finance or time-management. When stressed and overloaded with information, we tend to be influenced easily towards decisions we would disagree with if we had the right information and time to think. Although goals may be set, they are pushed aside by other pressures that are more short-term. Poor decisions cause poor health (and also poor mental health and other negative outcomes).

The project is funded by an UnLtd small grant through the University of Manchester Social Enterprise innovation initiative:

Why is this new?

There are lots of apps out there (see for example http://www.businessinsider.com/functionality-is-key-to-the-future-of-health-apps-2014-6), but most are targeted towards individuals as “consumers”, and do not address the more challenging psychological and social problems. Usually they are just addressing a single issue in isolation, such as exercise. Although Google and Apple have started to develop “health platforms” (http://www.businessinsider.com/google-fit-apple-healthkit-2014-6) they are mostly concerned with integrated sensors and tracking of physical health, not with psychological health.

Some initial requirements

These are the broad requirements that I am starting with:

  1. Citizen led: the users should be in control of both the technology and the process by which they improve their decisions and action.
  2. Promoting reflection and reasoning: reflection and reasoning are high-level cognitive processes that can put the user in control because they involve awareness, deliberate goal setting and planning. This contrasts with behavioural “nudging” which affects decisions unconsciously and have been criticised as unethical (http://www.newscientist.com/article/mg21228376.500-nudge-policies-are-another-name-for-coercion.html#.U-ZXsvldVmU). For this reason I think the app should be called a “cognitive assistance” app.
  3. Promoting social support: Decisions are influenced by the wider social context. Therefore the app should not just be helping individuals in isolation, but must take account of their social environment and promote social support.

Some technical and research challenges

One way to help people resist pressures to make bad decisions is to raise awareness of significant and relevant information (such as why the original goals were set in the first place) and to encourage reasoning. In particular, this might include the following:

  • Smart prompts and reminders – sensitive to the person’s mood and circumstances. Users could also be prompted to send encouragement to others when they may need it.
  • Visualisations to draw attention to the important things, when a user is overloaded with information and options.
  • A kind of dialogue system to help with reasoning (not necessarily using text)

Getting these right is extremely challenging. In particular, the app must be sensitive towards the user’s state of mind and changing context. Smart sensor technology can help with this. (See for example, the MindTech project at http://www.mindtech.org.uk/technologies-overview.html). The challenge that particularly interests me is the modelling of the user’s mental attitudes so that the reminders are not disruptive or insensitive. To support reasoning, some principles of online Cognitive Behavioural Therapy (http://www.nice.org.uk/Guidance/ta97) may be applied. In future posts I will discuss the role of theories and models, as well as the participatory software design process.