This article is part of our special series on Flesh

Our perspective on how to anchor emerging technologies in a deep understanding of our sensory experiences. By Ian Dull and Maria Cury

The way we imagine the future of technology usually focuses on our eyes: the screens we will look at and where these will be placed. Much of the imagery about future virtual worlds – from science fiction to Silicon Valley – has focused what we can see: whether in Ready Player One, Tron, or today’s ideas of the metaverse, we imagine ourselves stepping into an immersive environment where everything we see is in an alternate digital space, our physical bodies left stumbling into tables, chairs, and walls in an uninteresting room.

But what about our other senses? From spatial and augmented audio to haptic gloves to explorations in virtual scent, many technology and consumer electronics players have been exploring ways to replicate and augment our traditional senses.

And yet, in our experience working with technology companies, we have encountered teams struggling to innovate on senses – especially beyond sight – and grappling with a common set of questions, in particular when it comes to:

Defining an overall perspective on augmentation that is both commercially and humanly relevant: Many teams we have encountered struggle with what commercially successful augmentation of senses beyond vision should feel like. Should augmentation be ‘better than’ the human average, or just deeper digital immersion? What are we augmenting in hearing or touch, and why?
Identifying meaningful use cases: Designers and engineers dream up inventive use cases for emerging sensory technologies. Yet which of these are likely to be compelling, differentiated, and meaningful to people? What should be prioritised in order to drive wide and long-lasting adoption of a new technology?
Understanding what makes an experience ‘good’: What is subjectively experienced as high quality can be a source of confusion and mystery. What are the metrics to follow to deliver great perceived experiences?
Anticipating barriers to adoption: The history of science and technology is cluttered with examples of supposedly revolutionary products – from Betamax to Google Glass – which totally flopped. What might keep people from incorporating a new technology, particularly a corporeal one, into their lives?

Not having a people-centered approach to the senses can lead to innovations that are either gimmicky in-the-moment delights on the one hand, or attempts to automate and replace what we do with our senses to make life easier. But these ways of innovating on the senses in technology might not reflect how people value and use the senses in the context of daily life.

A different approach

ReD has been exploring the intersection of the senses and technology, using anthropological approaches to deeply understand the experience, affordances, and values of our senses and the possibilities new technologies offer expanding them. Our work has led us to another approach: providing super-human sensing. This means enhancing and supercharging already valuable human skills, perceptual capabilities, and experiences in the physical and virtual worlds.

A better understanding of the human value and use of the senses in everyday contexts can help with product definition, capabilities and differentiating experiences to invest in, and anticipating the biggest barriers and drivers of tech adoption. What, then, does it mean to deliver superhuman senses, and what models can technologists draw on in practice?

Across our work, we have observed:

Our senses contain clues to what is valuable to enhance: The way we hone our senses – either through active training or gained experience over time – reflects the skills or talents we value: The supertaster. Eagle eyes. A nose. Golden ears. Perfect pitch. Our societies have encoded countless ‘superhuman’ abilities into our language. Defining what abilities matter and will be meaningful starts with understanding what is valued today around the senses, and in what contexts. Societal values around the senses create tension with technology that strives to automate or replace sensory capabilities, and suggests that high value use-cases are ones that amplify rather than replace human skills.

Our senses offer a more immediate interface with the world: Visual processing – often considered the way of taking in information – is just one particular type of input, with unique affordances and biases. Our other senses, however, offer ‘interfaces’ collecting other kinds of ‘data’ about our surroundings. Hearing offers 360° awareness: we hear and interpret sounds in our environment without actively listening. Our skin captures both focused inputs – the localised and granular sensations of touch – and diffuse ones – as with sound vibrations or temperature. Smell is hyper-attuned to changes in our environment and enables us to tap into emotional states from memory. Selecting the right sense or adeptly combining them can be powerful, such as the euphoria and trance-like state people often describe with electronic music. There is an opportunity for personal devices of the future to tap into these alternative interaction paradigms, to help us experience worlds and information in new ways.

Our senses help us assess whether something is ‘real’ or not: Whether it’s reaching out in front of us to check if an illusion is really there, or sniffing a material to judge if it’s really leather, we use our senses to ground our truth and reality. In designing for a virtual world, which aspects of our sensory experience matter most to keep us absorbed in a world and feeling immersed? By understanding the aspects of context that are critical to making a world or an experience feel real, credible, or legible among seemingly endless choices, there is an opportunity to streamline and prioritise what to render in high-fidelity for the senses.

How to build for the senses

While there is no one-size-fits-all strategy for the many different senses, technologies, and products being developed, we have used three key methods to identify meaningful and differentiated opportunities within our sensory experiences:

Embody the expert

People often struggle to articulate how they experience their senses and, even more, how their senses could be enhanced. The richest learnings, thus, emerge at the extremes: people who have honed extraordinary sensory abilities – or who face significant hindrances to their senses – reveal the most developed vocabularies, behaviours, and understandings of the senses. By observing how these “super-sensers” experience their domain of expertise, as well as challenging them to apply their skills to new contexts and domains, we can learn an extraordinary amount.

Go in-situ

How we use our senses – and where individual senses are foregrounded or backgrounded –- is situated in particular contexts: even experts aren’t aware of how and when they are relying on particular senses, combinations of them, or particular cues they are drawing from the environment. To develop the kind of robust understanding needed to engineer new product and platform capabilities, we need to observe how the senses are activated, prioritised, and elevated across different contexts. It isn’t enough to rely on the already-prioritised use cases for a technology: the full value and patterns of using our senses often come through in extreme situations or in activities adjacent to these use cases, like communicating at a loud concert for hearing. Mapping a range of valuable situations to observe from a human perspective can make all the difference in identifying a differentiating, new capability.

Observe both macro and micro

We cannot only understand the detailed everyday, practical actions around a given sense; we need to understand the macro value of a sense, too – i.e. what kinds of activities are broadly valuable across society to enhance – in order to focus on real commercial opportunities and addressable markets for a new product. That comes from zooming out and looking at how a given sense or sensory skill is constructed as valuable in society or in communities today, and what kinds of systems of value (e.g. economic, creative, caretaking) are used.. Moving from the macro to the micro, detailed observation can then help determine what behaviors in particular are valuable to enhance and how.

Emerging technologies offer fascinating possibilities for new sensory experiences, but we risk these being irrelevant or off-putting if we do not anchor sensory technologies in an understanding of how people construct, value, and experience a given sense today. By looking – or rather, listening, touching, tasting, or feeling – closely and contextually, we can build towards technologies that meaningfully enhance some of the most sophisticated sensors that exist today: our senses.

This feature is part of our special series on Flesh

Featured

Do bodies still matter in the age of AI? – a Live podcast