On Fabrication: Evidence, Truths, and Design. Uses and Conditioning of digital Representation Tools

Émile De Visscher

Dear Simone and Francesco, before getting to your current practices, could you remind us of your background and education?

Simone Niquille

One’s path always makes more sense with hindsight. I went to the Rhode Island School of Design to study photography, but upon finding out that they got rid of their wet labs, I went for Graphic Design instead. Being fascinated with branding, identity, and representation­al politics, I ended up doing my Master’s thesis on the value of faces with the advent of digital photography and online distribution–how ‘real’ people’s faces end up in spam emails and fake Facebook chat pop-up windows. The work further investigated facial recognition technology and its scientific robustness by working with celebrity lookalikes and creating a series of obfuscating t-shirts inspired by dazzle camouflage. Ultimately this work led me to investigate the processes behind facial recognition and computer vision¹ more broadly, which required a deep dive into computer-generated imagery. I guess we can get into more details about the correlation between these fields later on...

Francesco Sebregondi

I was trained as an architect in Paris and graduated in 2008–right when the financial crisis was putting the global construction industry to a halt. I guess I was part of a generation of architects that did not really have another choice but to explore other ways of doing architecture. I moved to London to attend a Master’s degree at the Centre for Research Architecture at Goldsmiths supervised by Eyal Weizman. I was lucky enough to join in at a time of intellectual and political effervescence at the CRA and, more generally, at Goldsmiths. The Forensic Architecture project started in 2011, and I joined its first core team. I worked as an FA researcher and project coordinator for ten years, with a focus on Palestine-oriented investigations. This research practice largely informed my PhD, which studied the security architecture of the Gaza blockade. I moved back to Paris in 2018, at a time when tensions between police forces and social/political movements were very high. I realised that I could leverage my experience in digital forensics to contribute to the growing demand for accountability from State forces. After leading two FA investigations on prominent cases of police violence in France,² I founded INDEX in 2021, an independent expertise laboratory that continues the work of FA in terms of investigating allegations of State violence, while grounding it in the French context. Fig. 3

EDV

It seems that both of you have explored, via different paths, the use of digital tools, in particular 3D visualisation. How do you see the virtual world and the construction of digital spaces? Are they simply new ways of visualising and prefiguring a project (like drawing has always been for architects and designers), or do they raise more fundamental questions?

FS

I don’t use the word “virtual” to designate digital spaces and processes: the digital is an integral part of our contemporary reality through the way we experience it and make sense of it. With Forensic Architecture, we began re-using tools that were traditionally used to design and prefigure architec­tural projects, in order to study and reconstruct events that had already taken place: the bombing of a neighbourhood in Gaza, a drone strike on a family house in Pakistan... Case after case, this set of ad-hoc techniques and methods has begun to form some kind of new architectural/media discipline: more concerned with a critical analysis of lived-in environments than with their design. I believe this form of active, situated critique also participates in transforming environments.

SN

In my practice, 3D software is a tool and a site of investigation in itself. A current strand of work is focusing on computer vision training datasets in the context of indoor navigation. This visualisation technology is primarily developed for domestic use in what has been marketed as the smart home–think automatic vacuum cleaners, doorbells with cameras linked to your smart-phone, baby monitors and robot assistants. Some examples of existing products are Amazon’s Astro household robot, Nest monitoring cameras and iRobot’s Roomba vacuum cleaner. What fascinates me about these specific devices is that they face a data collection problem due to accessibility: homes are supposed to be private spaces, how can thousands of visual data be collected from people’s homes? One answer has been synthetic data, meaning 3D models of floor plans, furniture, and products that are assembled into ‘model homes’. Still images of these digital arrangements are then rendered by calculating random camera trajectories moving through these ‘spaces’. Fig. 1 Fig. 4 These still images then become the training dataset. Architecture and product design both use 3D modelling in the design workflow.

Using this data to construct digital spaces, and assuming that they do represent a domestic environment is inviting, and an efficient solution. Yet, I argue that this synthetic data ignores the ambiguity and complexity that makes a space a home, a personal space, while also struggling with politics of categorisation, data availability, and language. In one of the datasets I have researched, a listed category is ‘guns’. Is that a domestic category? Is this word referring to a toy gun or a weapon? Furthermore, the category ‘chair’ included 3D models of a toilet, wheelchair, baby stroller and electric chair. The inclusion of the electric chair is particularly shocking. Creating these datasets isn’t a computational process but one driven by human choices. It also isn’t a question of computer sciences but one that addresses philosophy, politics and culture. By framing this as a computational problem, domestic spaces become generalised (digitally) and reduced to the available 3D models. Here, generalisation is a functional desire that considers that the technology should be implemented in products sold globally. Once presented in categories though, with the 3D model’s source and context removed, the assumptions made at the point of data assembly suddenly present as ground truth. In the case of synthetic computer vision training datasets, digital models directly influence how an object will navigate, interpret and interact with the world. For such technology to function properly, domestic space needs to mirror the digital model. An interesting shift in that regard has been Ikea’s slow transition to a fully rendered catalogue rather than photographed scenographies. In order to create their digital imagery, Ikea created a huge dataset of all their products including ‘life assets’, objects that will make a room look as if someone lives there, with food, a cat, etc. Directly training a vacuum cleaner on an Ikea dataset, for example, seems like an effective answer to creating computer vision that will navigate indoor spaces and recognise furniture correctly. However, it is a seamless solution to a world that is (thankfully!) much less standardised. I can buy an Ikea chair but I may chop it apart and use it as a bookshelf. What might sound like brute force obfuscation tactics against computer vision is actually daily life. In making these synthetic datasets visible, my practice aims to highlight the human decision-making processes within technology that presents as objective, challenging its authority.

EDV

You both work around, with and in forensics... At the end of the day, which positions does architecture–or the precise choice of representing a real scene through the calculation of a digital space–allow you to present as a researcher and as expert in the media or in court? Also, in what way can your practices–focusing on the question of representation and appearance of individuals in virtual spaces–lead to investigating evidence production within digital media?

FS

I think it’s safe to say that, over the last decade, Forensic Architecture has demonstrated how the field of knowledge associated with architecture – expanded through multiple media practices–can significantly contribute to the making of public truth. In our increasingly complex and atomised media landscape, architecture emerges as a tool to piece things together, to reas­semble fragments, to draw connections between them. Retrospectively, it seems clear to me that our investigative work has remained, quite literally, an architectural practice: forensic architecture–intended as a practice which now extends beyond the work of the eponymous agency–leverages the synthetic power of architecture to produce meaning and coherence out of messy fields of data.

There can be no society without shared truths holding it together. Considering that among those who worked to dismantle society as an idea, some are still in power today, it is no surprise that the very conditions for the emergence of shared truths have come under serious attack in recent years. In this context, any effort to rebuild a common ground for an informed and inclusive public debate to take place on societal issues, acquires a strong political character.

The investigative research conducted at FA is always grounded in particular events and situations. There are two reasons for this. First, the project is committed to seeking truth and justice in a very practical sense– namely, case by case. Secondly, understanding the details and specificity of a case provides a valuable entry point to engage with the wider issues that collide within it. With INDEX, we are pursuing the same method, pushing this grounded approach one step further by focusing on cases of police violence in France. The issue of police violence has turned into a profoundly divisive debate here, and its treatment in the public sphere tends to reduce it to a matter of a priori opinions about the police. In our work, we focus on establishing the facts around police operations and activities–once again, case by case.

SN

The history of 3D avatars is intrinsically linked with the history of forensic sciences. The first 3D avatar ‘Jack’ was created by Norman Badler at the University of Pennsylvania’s Center for Human Modeling and Simulation. The virtual figure was based on anthropometric (body measurements) of US Army personnel, collected as part of the US Army Anthropometric Survey (ANSUR) of 1988.³ The survey’s intended use case was a design guide for clothing and equipment. Referred to as a ‘virtual human being’, Jack the avatar was developed to analyse ergonomic design of airplane cockpits and went on to be part of the ergonomic design software program of the same name that has ultimately been integrated into Siemens’ software portfolio. Jack the software program is used to, for example, analyse the ergonomic design of a factory, the individual workstations as well as the overall layout. I have written about the politics behind these anthropometric datasets and ultimately how they are integrated into the design of ergonomic software as a drop-down selection for ‘body-type’. The problematics echo in part the critique of forensics techniques that attempt to render subjectivity as objectivity by translation of corporeality into numbers. Measurements are not objective. They have a context and subjectivity that cannot be generalised. The use of anthropometric measurements in bertillonage, for example, ascribing criminal tendencies to individuals based on repeating patterns,⁴ is similarly violent and problematic in that it reduces people to data, easily confused with scientific fact.

EDV

“Measurements are not objective” is a really interesting statement in regards to the work of Forensic Architecture and INDEX. Francesco, when you were affirming the current need to build collective truth, aren’t you also using specific measurement tools and visualisation techniques, generating a specific lens on facts? Aren’t your tools as subjective and predetermined as the ones of police forensics? If we take journalism as an example, the use of digital technologies in legal or journalistic investigations seems to be increasingly common–what kind of political role does this entail, and how is journalism dealing with digital technologies?

FS

There are several ways of answering this question. On a pragmatic level, the answer is no: our investigations and reports are primarily concerned with establishing what Hannah Arendt calls “factual truths”, namely, truths that rely on material evidence.⁵ When we establish in a 3D model the precise location of a police officer at the moment he fires his gun, based on video footage capturing the scene, and measure the distance from his target, for example, I think we are undertaking an objective measurement. In fact, the 3D reconstruction of a scene captured in a two-dimensional image–which is a recurrent process in our work–is also an effort to move beyond the subjective bias of a specific lens, frame, or perspective on a given event.

On another level, like all sciences, forensics has a political history, that is tightly intertwined with the history of State surveillance and control over its population. When it comes to watching, tracking, and measuring people and things, the State has historically had a near-monopoly on the means and capabilities to do so–and that is also a reason why forensics has come to play such a central role in contemporary policing. Recent technological shifts, such as the popularisation of camera-enabled smartphones or social media, have opened a few breaches in this monopoly on forensic means. With FA or INDEX, we set out to exploit these breaches to produce counter-forensics. In this context, reconstructing factual truths through an objective analysis of the material evidence available does not imply political neutrality, quite the contrary: it means challenging a fundamental asymmetry of power.

SN

From a design research perspective, my use of software is applied research. How is the interface structured? What is the language used to describe a software program’s menus, buttons, options? What is the intended use of a software program and how does that influence its function? I believe that this level of scrutiny should apply to software programs used in forensic investigations. Software might influence the way a visualisation is created. A software program isn’t an objective tool but rather a written code formulated by someone. What matters is who this someone is and what their intentions and politics are...

An example that comes to mind is the 2015 ban of RAW photographs by Reuters. The news agency requires journalists to submit .jpgs or .tifs compressed in camera to ensure that the photographs have not been tampered with, citing ethics and speed as reasons for the policy change. Quoting Reuters: “As eyewitness accounts of events covered by dedicated and responsible journalists, Reuters Pictures must reflect reality. While we aim for photography of the highest aesthetic quality, our goal is not to artistically interpret the news.”⁶ Yet it is a mistake to assume compression algorithms are objective. Photography has evolved from the digital to the computational. Arguably computational photography is closer to computer-generated imagery than the word photography implies.

EDV

We tackle here the core of this article’s theme: the possibility to fabricate truths through imaging technologies. What is the relationship between fabrication and proof? Also Simone, can you tell us more about this idea of variable truth you seem to be working on, by mixing immersive experimental projects with longer-term explorations around the particularly intriguing question of “parametric truth”?

FS

There is no doubt that images can lie, even more so when it comes to digital, synthetic images. We don’t expect our audience to take the truthfulness of our visual reconstructions for granted; in fact, we strive to use images’ inherent power of conviction with great care. What is important to us is to present digital reconstructions not as a mere result, but as a process. In our reports, about 90% of the content is dedicated to exposing how we proceeded to reconstruct an event, step by step. We are quite demanding with our audience: we ask viewers to adopt an active and engaged position when looking at our work, rather than addressing them as passive receivers of our authoritative conclusions. This full disclosure of the “making” of an image is for us the best guarantee against the risk of falsification or distortion of facts, at the same time as it provides a rigorous frame to use the power of visual analysis and representation for the production of evidence.

SN

Parametric truth isn’t a variable truth, but rather one relying on software as justification. ‘I’ve input this into the machine and this other thing came out. Therefore, it must be true.’ Say you only know points A and C of an arms movement; inputting points A and C in an animation software program won’t reveal point B. It will produce a possible version, but not a truth. Simply because something is calculated/computed, does not make it factual nor objective. With my project Parametric Truth, I am investigating software used to create ground truths, be that in forensic animations or the creation of synthetic training datasets for computer vision. What might sound like differing fields of research actually converge in using the digital space as an environment to reflect on/make decisions about the world. One that is alive with, and because of, ambiguity. Software does not compute ambiguity.

EDV

The terms “truth crisis” and “post-truth” are, of course, central to your research. How can your approaches overcome the debates and the status quo linked to the fact that everyone can express themselves on facts and interpret them, or even build “alternative” facts? Is there, in your opinion, a way specific to design that would allow us to get out of a direct confrontation between conspiracy theories and fact-checking? What to do in the face of the relativist position?

SN

Context matters in the realm of data. By itself data might appear factual, but disconnected from its context it lives a life of free association.

FS

What has been named the “post-truth” condition corresponds to a crisis of the traditional means of production of public truth. The development of social media and the resulting, exponential multiplication of sources of information circulating in the public sphere are certainly playing an impor­tant role here. And as previously mentioned, there are powerful agents out there actively trying to stir up this crisis: indeed, individuals isolated in their belief and opinion bubble are arguably easier to rule over than a society united around shared truths and a common consciousness of their reality. That said, the response we strive to formulate with projects such as FA or INDEX, among many others, does not imply reverting back to the tradi­tional truth condition by reinstating the authoritative power of centralised truth institutions–such as the judiciary, the university, or mainstream media. As Eyal Weizman has been arguing in recent years, the ongoing decentralisation of the means of truth production also presents great opportunities for a wider access to public truth, and thereby, for collective emancipation. For us, an appropriate response to the current crisis is rather to develop collective and inclusive practices of truth production. Such a prospect involves disseminating and sharing tools within civil society that can sharpen our common critical gaze.

EDV

Could you detail the associative model of INDEX? How do you collaborate with civil society?

FS

As far as democracies go, France is one in which civil society’s role has historically been relatively weak, compared to other western countries. The State remains in charge of much of the public affairs, and a number of its institutions are only partly accountable to the public: for example, the IGPN– the General Inspection of the National Police, responsible for controlling and sanctioning its actions–is not an independent body but is directly under the authority of the Director of the National Police. We established INDEX as a non-profit “association”–thereby anchoring it firmly within civil society–with a view to demanding more transparency and accountability from a public institution such as the police. Being grounded in the local context of the cases we investigate enables us to join forces and build durable links with other civil society actors, collectives, and organisations that have been engaged in this battle for accountability for a long time already.

EDV

Both of your practices deploy deliberately non-realistic aesthetics of bodies, spaces, and gestures. Why is it important? Is there a desire to schematise, or is it simply a question of means? Why not make photorealistic images? In your experience, what is the main purpose of using 3D reconstructions: revealing a hidden truth? Representation for better understanding in court? The possibility of appealing to a wider audience?

FS

We were discussing this point (non-realistic aesthetics) with Simone the other day. With FA, and now with INDEX, we deliberately choose to produce digital renderings of our 3D reconstructions that are schematic. That is because the model is not the event, just like the map is not the territory: their respective role is to produce a particular understanding of the wider complexity they capture. It is important for us to produce images and representations that explicitly underline their analytical character and don’t communicate a false sense of event reproduction “as it happened”.

In our work, 3D reconstructions are used first and foremost as an investigative tool: among the criteria we use to choose our cases, one is whether there is a spatial problem to be “cracked” through a detailed 3D model of the event. With regards to the narrative power or immersive dimension of 3D reconstructions–especially when delivered in a video format–they can certainly be very effective in communicating complex conditions in a synthet­ic and concise form. That’s the second reason we are interested in using 3D reconstructions: publishing a report in a 10-min video format instead of a 100-page PDF also means ensuring that its content can circulate more broadly within the public sphere. It is a matter of taking responsibility for the public engagement we expect around our work.

SN

‘Photorealism’ is a misleading term. It suggests that photography equals reality, which is then used as a benchmark for computer-generated imagery. CGI has more interesting use cases than imitating reality in the first place. I like the term ‘mundane render’ instead, which appeared on a Blender forum thread sometime in 2020. An Ikea catalogue is a mundane render, the image’s main objective is to not attract attention. Such renders contribute to maintaining normativity. In my work, I am drawn to familiar references in order to communicate concepts that are either foreign or so obvious that we stopped looking into them. Pop-culture is something I borrow from in my visual and audio work. In the film The Fragility of Life I worked with Teresa Barnwell, a Hillary Clinton impersonator, documenting her days leading up to the 2016 US presidential elections. The film is a combination of live footage and 3D animation. We 3D scanned Teresa to ask legal questions on data ownership. Does one own the right to one’s own 3D mesh? It is a powerful bit of data that can be animated at will. One scene in the film shows the 3D scan of Teresa decomposing. Fig. 5 Fig. 6 Technically I am lowering the mesh’s resolution, its polygon count, which results in a very jagged-looking non-human form. Rather than using it as an effect, I like to directly use the way CGI is constructed to reveal its mechanism–not simply to expose, but also for its beauty.

EDV

Simone, you mentioned in a previous publication that “[c]orporeal data does not execute violence on a body by simply existing, but its mechanism is intertwined with economic incentives of optimisation and safety. In the case of ergonomic simulation software, as described previously, worker safety plays a large factor in insurance policies and as such is an economic interest beyond production efficiency.”⁷ How do you study, detail, and give form to these political implications? Could you say that through your installations and performances, your intention converges with Francesco’s, that is, to create public engagement, but that it also aims, in your case, to better understand the hidden mechanisms of algorithms and their impacts on our lives?

SN

As a design research practitioner, I am used to asking questions, inquiring into software, case studies, data. My output tends to be in writing and film as it allows me to communicate with different audiences in varying aesthetics and tones of voice. A piece of writing often holds detailed references and research, while my film work is more of a contemplative and hopefully humorous space, for the sake of accessibility. The subject matter can easily tip towards a dogmatic tone but I believe this tends to shut down communication. Visually borrowing from pop-culture and working with voice-over participants of all ages, languages and accents is one way I try to create a (visual) space for reflection. Exhibitions and publications have been a productive space to do this research, but I am constantly looking for collaborators in other fields. Connection across and beyond expertise is necessary for this work. I like to think of research as crosspollination, specifically research on computer vision which touches on topics of photography, computer science, philosophy, ethics, etc. The bigger picture is to compile a critical history of computer vision that ties in elements of parametric truth. An aspect that Forensic Architecture pioneered is the combination of aesthetic practices and social impact. The aesthetic output of my practice is important but as a next step, being directly involved rather than criticising technology from what sometimes feels ‘afar’ is what I am looking towards. Specifically, computer vision, as it reveals the bias and individual interpretations of the world that are within each of us. I think people, for the most part, are aware of their specific worldview, yet the narrative around technology proposes otherwise, or to take it a step further, it presents as a solution to this subjectivity. Photography went through a similar struggle between being considered an artistic medium and a forensic science. Computer vision then is a contemporary technological extension of human’s interpretation of the world, or more accurately, the worldview of a few computer scientists, based on available data.

EDV

The questions of truth and proof are crucial in your explorations. How do you qualify these reconstructed truths? Is there no contradiction between fabrication and truth?

FS

The first book we published with Forensic Architecture–an anthology of our investigations over the first four years of the project–was subtitled: “The Architecture of Public Truth” (Forensis, Sternberg Press, 2014). By that we meant to affirm that truth is never evident, never given, but is always mediated. Truth can only emerge in the public sphere through argumentation and negotiation–whether they take place in judicial, political, or media forums. As such, public truth is, indeed, the result of a constructive process.