Physical Computing Final – Fireflies

My project was built in collaboration with Abby Lee and Tianxu Zhao. Together we built “Fireflies”, a concentration game utilizing a Muse EEG, p5 animation visible through a projector, and a Bluetooth enabled jar trigger. The process would be that once our user opens the jar, our animation of fireflies will appear. When our user is prompted to focus, the Muse will transmit that our user is in a state of concentration, prompting the flies in our animation to concentrate in conjunction. We see this game as an interesting jumping-off point for many neuroscience-related conversations.

I have had laboratory experience working with an electroencephalogram, which read the presence and absence of polarized oxygenated blood as electrical frequencies. Different types of frequencies, also known as neural oscillations, occur simultaneously all over the brain, and their functions vary depending on which region of the brain they take place in. We want to be clear with our audience – Muse is not a scientific apparatus. In fact, the company misrepresents many of the types of data the EEG reads and the inaccuracy of that data. That being said, Muse is capable of reading very localized, very simple frequency and amplitude changes that are understood to be associated with the presence and absence of concentration. Concentration in it of itself is not an entirely well-understood phenomenon. However, at a reductive level that reflects the Muse’s limitations, our animation is contingent on a change from lower frequency, lower amplitude oscillatory behavior (usually referred to as alpha) which is commonly associated with resting state, to a higher frequency, higher amplitude neural oscillatory behavior (usually referred to as beta) which is commonly associated with concentration.

We integrated our Muse to our p5 animation through a series of steps. First, we read the Muse EEG data with a third party mobile application titled “Muse Monitor” on one of our phones. Our phone then transmits the live EEG data using Open Sound Control protocol to a processing sketch. We can then input the OSC feedback from our processing sketch to our p5 animation locally.

We connect our jar to our animation using an Arduino Nano, which has a built-in Bluetooth transmitter. We have programmed this Nano with a proprietary Bluetooth transmission library, capable of easily interfacing with our local p5 animation. Our Nano is attached to a button, which is triggered by the removal of the jar’s lid.

The game plan is that when the lid of our jar is removed, our p5 animation will play, allowing the Muse EEG feedback to be represented. Once the lid of the jar is replaced, the animation stops. To be clear – the Muse and jar operate independently of one another.

Ultimately my group learned how to take Muse data and use it in a p5 sketch. We learned how to connect an Arduino to a computer via Bluetooth. We discovered that the delays created by Neopixel code can interfere with Bluetooth signal transmission. We broke a few Arduino Nanos trying to figure out an optimal long-lasting power source for our jar.

Special thanks to:

Jeff Feddersen, our Physical Computing professor, without whom we could not have realized this project, let alone use a multimeter.

Louise Lessel for pointing us to Gene Kogan’s OSC processing library on GitHub, and being an approachable and accessible wealth of Muse knowledge.

Tom Igoe for teaching us how to connect devices with Bluetooth.

Residents Jesse Simpson, Aiden Nelson, Ellen Nickles, and Anthony Bui for their help in troubleshooting.

Playtesting a prototype animation
Final prototype demonstration

Pcomp – Revised Prototype

This week our group re-evaluated our project concept based on user feedback from our previous class. Our users made two concepts clear:

1. The metaphorical connection between fireflies, fire, and personal/psychological connection was not entirely clear

2. It is not universally intuitive when given a jar to try and catch fireflies

Based on this feedback, we decided to refrain from integrating a Microsoft Kinect to track our user’s body motions. We made the following progress with our animation, figured out how to integrate our muse into our animation, and made steps in building our physical jar process trigger.

Pcomp – Paper Prototype Prep

This week, my group was tasked with developing a project description, a system diagram, timeline, and bill of materials in preparation for our playtesting.

As a group, we constructed our final narrative and drew out the necessary components provided in the images below:

Essentially, we will have 3 components linked to a p5 sketch – a Neurosky EEG reader, a jar that can light up and vibrate, and an Xbox Kinect camera. Our animation will begin as a flame with a dark background. When our user twists off the top of the jar, the fire will extinguish, and turn into smoke. Depending on our user feedback, our user will be prompted to focus on the smoke, and it will turn into an avatar for our user, as fireflies begin to emerge from the darkness. With the jar, registered by the Neurosky, our user may capture up to 10 fireflies. For every firefly caught, our jar will become brighter and vibrate. Once the user has caught 10 fireflies, the animation will revert back to a fire, and the user will reset the animation by replacing the cap on our jar.

We decided that we would determine our floor plan based on user feedback.

Our group had been working on dynamic timeline and bill of material documents, that we have made accessible on this page.

Pcomp – Initial Final Concepts

For my Physical Computing final, I have two potential ideas:

1. “One Track Mind”

One-track mind is a turntable, with an RC car set atop it. The car and turntable would be moving in opposite directions, at the same velocity, so that the car would stay in the same place on the turntable while it spins. The car and turntable would be in communication with one another through Bluetooth. The interactive, and most difficult part, of this project, would be a potentiometer that would change the speed of the turntable and car simultaneously. I am also considering integrating some sort of LED into the turntable that can help delineate what speeds the turntable/car has reached. I may have the RC car look like a brain.

This is a sketch of my project "one track mind" as described in the text

2. Title TBD

This project would utilize the portable EEG reader NeuroSky, or a more affordable model (pending feedback) to halt the perpetual motion of an RC car. For this project, I would definitely try to make the car look like a brain.

This is a sketch of my project, Title TBD, as described in the text

Pcomp Midterm – Numbskull

With my partner, Simone Salvo, I created a “talking” skull, which would play a Donald trump quotation sound-byte, display a corresponding fake tweet, and flash red, white, and blue eyes – all activated by a hand press on Dostoyevsky’s The Idiot.

In creating this project, we used Arduino Uno because it gives out higher voltage than our standard Arduino Nano (5V vs 3V). Our concept went through two iterations, one where we planned for the use of simple red LEDs for our eyes, and our later iteration where we used multicolor NeoPixel LEDs. Many of the components we programmed for we had worked with in one way or another in our lab, as such much of our code had essentially been pre-written on the Arduino side. However, we found that in order to make our button serial input trigger we had to go back and “uncouple” the code we had written, separating each individual action into their own function. In doing so, we realized that we could have reconfigured our button press feedback to rely on the buttonPress() function, instead of the newState()/oldState() functions. It is simple conditions like these that could have taken out multiple instances of delay() from our Arduino code – which we have learned could have been mitigated with millis().

An initial stage, where our mouth servo motor was successfully being controlled by our book-button.
Fabrication stage

In all, this project worked exactly how we wanted it to for our presentation – moving forward, we could do more to “clean up” our code, and utilize best practice. That is not to say that our code was poorly written, in fact, this project was an opportunity to appreciate the many ways in which solutions can present in programming problems. However, I have a new appreciation for writing code in a style that separates different functions distinctly.

Pcomp Creative Experiment – Home Made Pong Retrospective

Without realizing it, I went ahead in the week we were supposed to learn about the basics of asynchronous communication (week 6) and tried to set up a feedback system with two types of serial input (a week 7 concept). With the help of my instructor, I discovered that (unsurprisingly) more work was needed to read feedback from both of my potentiometers, which I intended to use to determine the position of my pong paddles. I found the Arduino code relatively straightforward to program, though I struggled (and still struggle) to animate two elements of a p5 sketch creatively. To realize my vision, I used my Arduino code but interfaced it with a java sketch written in processing, available here.

One troubleshooting phase video
My successful result!

Though I may not have succeeded in creating the animation I desired in p5, this project was a great entry point into troubleshooting with serial input, and figuring out the complexities of math-based objects in p5. It certainly solidified my capability in integrating serial input into basic objects- however, when I wanted to create objects that relied on serial input but were able to react to the ball element in my sketch, I struggled to make the paddles react to the ball.

Pcomp Creative Experiment – Audio Player Retrospective

As a creative adaptation to the fifth “tone output” lab I attempted to get a speaker to play a preloaded wav file of “The Price is Right” theme song. My hope was that I could use 3 buttons to operate as rewind, play/pause, and fast forward commands for my song. Though this project was not successful, I found the process of attempting to troubleshoot my circuit and code incredibly valuable.

This was my first attempt at assembling a circuit with an Arduino Uno, and it was a wakeup call for me to better understand C++ programming in the Arduino application. I was able to reach a point of comfort where I could test feedback from the components in my circuit. This was helpful in realizing that one of my buttons was broken, but that my other buttons worked and my SD card reader was transmitting information to my Uno.

I believe my failure came from my circuit building, and when I attempt this project again I will rectify two elements. First, I will try to build my circuit with different buttons, as I found these components hard to plug in and a source of frustration in my initial stages. Second, I was not using a “standard issue” speaker, instead, I used a cheap tweeter which I had purchased at Tinkersphere. I had checked this component, with a potentiometer, and while it “worked” I believe I should have soldered the comparatively frayed wire ends of the speaker to the more solid, easily pluggable wires.

Public Technology Observation – LinkNYC

In observing how consumers utilize LinkNYC, my cynicism and disdain for this public technology have been affirmed. On the surface, LinkNYC is benign, if not handy; built so that its buttons and touch screen sit at a height that is accessible to all, adults and children alike. As LinkNYC is a New York City initiative, Link booths are presumably ADA compliant. However, LinkNYC is a private piece of surveillance technology, and like all surveillance technology, it has a disparate impact on who is “targeted” by its true purpose. LinkNYC booths and their private parent company have been at the center of many lawsuits surrounding data privacy violations. The ease and inviting aesthetic of LinkNYC obfuscates its nefarious, similarly opaque practices of data collection and sale.

In my observation of how this technology is used, the majority of peoples who gave “links” more than a passing glance were individuals who appeared unconventionally housed. Were LinkNYC a public good, with no perverse incentive to collect and monetize the data of its users, I would take no issue with this dynamic. However, the fact that LinkNYC (in my observation) almost exclusively appeals to vulnerable peoples in our society, one must consider how features that may seem convenient at first glance, are ultimately predatory. For instance, LinkNYC has phone charging ports that, in keeping with the interaction trends I witnessed, unconventionally housed peoples utilized the most, and longest. Bear in mind, these are individuals who may not be able to charge their phones at home, at work, or even in a private business due to stigma – convenient design features for the more privileged can be lifelines for these individuals. 

The majority of people I witnessed using LinkNYC used these booths to charge their phones – these transactions took the longest, almost definitely contingent on the user’s battery charge.  I did not witness anyone use LinkNYC to place a phone call, perhaps this speaks to how little people place phone calls in general.

Creative Switch

For this assignment, I knew that I wanted to use a material that was readily accessible, unconventional, and conductive. Gatorade was a natural choice as a medium, as the same chemicals that imbue the beverage with electrolytic properties are what make it so beneficial for biological replenishment. The circuit sits atop the Gatorade bottle top, with the ground and live wires sitting inside the Gatorade bottle. The switch occurs when the Gatorade vessel tips and the Gatorade liquid completes the circuit between the two live wires.

In order to create this circuit, I learned how to solder. This was essential not only to connect my LED (circuit success indicator) to my resistor and power source. One of the greatest frustrations I ran into in this process was that I could not find a sufficiently small heat wrap to sufficiently waterproof my ground wires, in order to prevent my circuit from shorting. To mitigate this issue, I soldered the ground wires together and wrapped the soldered connection in electrical tape; passable, but not preferable.

I created two versions, the second more aesthetically pleasing. Below are videos documenting the first test of my circuit, the completion of my first design, and the completion of my final design.

Circuit Test
Switch Test Version 1
Switch Test Final Version