Project presentation


Project presentation
Project Mentor
​​​​Dr. Ellen Gethner

Team Members
Khalid Adkins
Zaid Alyasiri
Marco Rojas
Nicholas Stanford

Abstract

The goal of our project was to create a virtual environment where users could synthesize new sounding instruments, learn about music theory concepts such as key signatures, common chords, and how to play music. In our vision, the environment would be highly responsive to the qualities of the sound the custom instrument produces, as well as the user’s musical production. The user could play the instrument manually, using on-screen buttons, or add notes to a “sequencer”, a common interface seen in computer-generated music software. The programming language we have used to create the sounds and play notes is ChucK. ChucK is a strongly-timed language, an eloquent term used to describe the manner in which the virtual machine the language runs on plays sounds precisely when, and for how long, the program specifies. The language makes it easy to generate complex waveforms by stacking various frequencies on top of each other. 3ds Max, a 3d modeling software, was used to create 3d assets for our virtual environment. 3ds Max is user friendly and versatile, making it the ideal software for creating a visually appealing environment. Unity is the software development package that we used to create our interactive environment from the sounds we created in ChucK and the visual components made in 3ds Max. We applied the research we gathered on music theory to create a virtual environment that helps the audience learn more about music.

Introduction

Music and Mathematics, though highly intertwined, are rarely seen as such. Music is expressive, emotional, and tells stories. Mathematics, by comparison, seems logical and objective. Even so, music can be precisely described mathematically, and by observing the mathematics of music, insights can be drawn about the formation of key signatures, mellifluous chords, and the nature of the sound waves we perceive when we listen to an instrument. We hope this musical environment facilitates this type of insight. People can find new ways to think about music, and express themselves through the combination of art and science that is music.

We identified some characteristics of music and sound that can be reasoned about mathematically, and found ways to connect them to visual objects in the environment. We chose relationships that seemed intuitive. An example is the timbre of a sound (the “color” that is a result of the physical characteristics of an instrument or vibrating sound-producer) and the color or texture of an object. Another example is the frequency (or pitch) of the sound produced and the position of an object.

Theory Background(Include relative background and theory pertaining to the subject)

Digital music and audio has become more available to audiences in the last 20 years with the advance of the internet. We researched music theory in order to attempt to make software that could familiarize the audience with the concepts we encountered. The driving idea was to help users develop an appreciation of the distinction between music and sound. We want users to understand that when they hear a note, they are actually hearing a combination of frequencies, the superposition of which creates a precisely defined complex waveform. The user would learn these things all while engaging in an engaging interactive experience that would allow them to construct compositions with multiple parts playing simultaneously. We use the term, “compositions,” very broadly here, as users would be able to make melodies and rhythms, but should feel equally encouraged to simply make interesting sounds. In our software, they would do this using an interface that lets them define a frequency as the base and stack other frequencies, called overtones, on top of the base.

Methodology(complete description of all the work that has been completed)

Every week we did research and built components we believed needed to be added to our project as a whole. We asked ourselves ‘what is the difference between what we already know, and what we still needed to venture out and find?’ Then we searched for those components and found a way to implement them into our project as a whole. As computer science students, there was much for us to learn in the area of music theory. In one project we researched the creation of major scales, a fundamental concept in music, and applied visual elements to our project to help users play notes within a major scale.

The scope of our ambitious ideas required that our project use many different technologies. The software we used to produce computer generated audio was a programming language called “ChucK” that has unique characteristics that make it ideal for sound synthesis. The application used to model our environment, 3ds Max, contains no scripting, but it played a key part in our final project. We built our final project in Unity, a “game engine,” which we most readily adopted the use of because it was most directly related to our previous studies. A capstone project, for some, is a chance to show off what has been previously learned. On the contrary, nearly every aspect of our project required intense research and self-teaching, but we’ve persevered and built a final product to be proud of.

Results/Analysis (the results obtained from our methodology)

An important lesson we’ve taken from this experience is that the amount of forward planning, communication, and work required to accomplish a given objective is difficult, but these are certainly skills that can be improved. One primary reason for this is that unforeseen obstacles and complexities that demand a complete abandonment of previously completed work. Our initial natural model for a synthesizer was computationally prohibitive, and when it was deemed unusable, scrapping a large amount of code took a fair amount of humility. At the time, it seemed that time and effort had been wasted, but on the contrary, we’d become familiar enough with the technologies we’d been using to realize the errors in our implementation. In other areas, as well, the tasks which we believed to be the larger hurdles ended up seeming simple, while some simple tasks became painstaking. For instance, learning to code in ChucK sounded ominous, but effectively was accomplished within days. On the other hand, making a sound play when an object is clicked, and then continue playing until the mouse button is released, was a more daunting accomplishment than we had imagined. Nonetheless, our combined efforts resulted in our successful completion of the project. We created a new experience that allows the user to make and experience music in a new and unique way.

References

Atherton, J.; Wang, G. Chunity. “Integrated Audiovisual Programming in Unity.” NIME. 2018, June 3 - 6.

Cook, P. R.; Salazar, S; Wang, G. “Chuck: A Stongly Timed Computer Music Language.” Computer Music Journal. 2015, 39:4, 10-29.

Benham, C. E; Goold, J; Kerr, R; Newton, H. C. Harmonic Vibrations and Vibration Figures. Facsimile Publisher. 2019. 1909. Reprint.

Tymoczko, D. A Geometry of Music: Harmony and Counterpoint in the Extended Common Practice. Oxford University Press. 2011.