Responsive Environments: Designing Interactive, Sentient, and Intelligent Spaces (IMA NYU, Spring 2021)
Description
In this course, students focus on the study and development of responsive environments, framed within a contextual and critical exploration of the architectural space as a cultural, social and technological phenomenon, and also on the application of practical scenarios for interaction, sentience, and intelligence. Through the making of creative media designs and physical prototypes, students aim to demonstrate how our habitats/spaces/architectures can facilitate novel frameworks for experiencing and living.
The course aims to introduce students to a range of contemporary techniques on interactive and computational system development with a creative and speculative approach, using state-of-the-art development tools, such as Internet of Things, cognitive computing, as well as physical computing and real-time media design. The course consists of lectures (⅓), workshops (⅓), and practical sessions (⅓), and intends to provide a comprehensive and critical understanding both on the theory and practice of designing and implementing technologies for responsive spaces.
Assignments
Assignments are student-led. The instructor(s) will be providing comprehensive assistance and direction to ensure completed projects of a high quality.
- Homework Assignments: On selected weeks during the semester, you are asked to complete homework assignments (reading/writing, technical execution, and so on) that directly relate to the content covered in class. This is intended to help you build stronger skills with the associated tools and practices, and further advance your development for the course’s main project.
- Main Project: For the main project, you are asked to develop a scenario/system/installation of a unique environment that demonstrates responsiveness: interaction, intelligence, and/or sentience. Through prototyping, modeling, development, and refinement, you need to finalize a case study of a responsive environment on a physical, hybrid, and/or virtual domains.
Syllabus (in brief)
Session 1: Introduction to Responsive Environments & Sensing Systems. In this introductory session, we develop a solid understanding of the terms: “responsive environments”, “interaction”, “sentience”, and “intelligence”, familiarize with the associated context using historical, philosophical, artistic, and technological references, and extend the conceptual and theoretical content into practical applications using physical computing.
Workshop: This introductory workshop covers the basics of Arduino, including a demonstration of a basic sensing application. In addition, you will be exposed to a large pool of sensor interfaces, from which you have to select a certain combination that allows you create a unique sensing system.
Session 2: Installation Space / Phantasmagoria: In this session we focus on “installation spaces”, identifying origins and motivations related to installation and interactive art, analyzing and classifying properties that link to conceptualization, expression, and engagement, and exploring in detail ground-breaking works by reflecting on their media, development strategies, and final outcomes.
Workshop: This workshop demonstrates how to develop an addressable LED system with Arduino and the FastLED library. Other output systems will also be covered, such as actuators, motors, sound speakers, to name a few.
Session 3: Practical / Tutorials
Session 4: Telematic Emergencies. This class investigates the terms “automation” and “emergence”, and their relation with the field of Responsive Environments. Architectural, artistic, and design applications that link to automation practices are further being explored. Moreover, we view automation as a process to the development of systems (interactive, computational, perceptual) that become emergent, while reflecting critically on trends, motivations, opportunities, problems, and associated risks.
Workshop: The workshop covers an introduction to Node.js and the MQTT protocol. During the workshop, the development of a communication system is demonstrated, which utilizes a client to send and receive information to and from the main server.
Session 5: Shifted Dispositions. This session investigates (interior) spaces we inhabit and occupy, by utilizing a range of lenses and scales; from micro to macro, considering the wide array of invisible matter we are surrounded with, from molecules, bacteria, frequencies, disassociated properties, behavioral randomness, planetary informatics, and so on. The main objective is to realize the richness of the matter that co-exists with us at all times, and consider it as an inspiration and resource to the development of environments that are aware and expressive.
Workshop: In this workshop you will learn how to utilize Node-RED (make an IBM account here), JSON, and associated libraries to extract information from online databases and APIs.
Session 6: Middleware Interfacing . By diffusion, we characterize the spatial distribution and “disappearance” of the technological infrastructure in the interior space, assisting automation, media design, and/or artistic practice. Here, we analyze frameworks and practices that define approaches used in the development of environments that are responsive and intelligent, embedded into “the fabrics of everyday life”.
Workshop: This workshop focuses on the analysis and implementation of communication practices with clients, servers, databases, and APIs. With the use of middleware and related development techniques, we will see how to create dynamic configurations on diverse interactive systems.
Session 7: Mini Presentations
Session 8: Affective Environments & Cognitive Agents. With the use of machine learning, artificial intelligence, computational algorithms, and psychological models, a range of behavioral and emotional predictions can be estimated. By analyzing media inputs (i.e., images, sound, text, etc.), computational applications can demonstrate affective and cognitive responses. In this session we review these processes and reference related applications and creative practices.
Workshop: This workshop covers the implementation of multiple IBM Watson services, used for cognitive computing. We will see in detail how to create the services, and how to allow them communicate with our systems using the MQTT protocol.
Session 9: Practical / Tutorials
Workshop: Virtual Interactions. The workshop covers the basics on creating interactions with Unity, Arduino, and communication systems (such as serial, MQTT, and HTTP).
Session 10: Practical / Tutorials
Workshop: Achieving Interaction, Sentience, and Intelligence. Utilizing the IBM Watson SDK for Unity, and in particular the Speech-To-Text and Tone Analysis modules, we will extract sentiment tone of realtime speech and visualize it with the VFX Graph.
Session 11: Practical / Tutorials
Workshops: Unity VFX / Materials / Lighting / HDRP / ShaderGraph
Session 12, 13: Practical / Tutorials
Session 14: Final Presentations (Summative Assessment)
