Audio-Visual Interactive Synthesis and Sequencing

An environment made for the production of new sounds and music

FINAL PRESENTATION

AUDIO VISUAL EXPERIENCE

In this video we Demo the combination of audio - graphics in computer science.

Project presentation


Project presentation
Project Mentor
​​​​Dr. Ellen Gethner

Team Members
Khalid Adkins
Zaid Alyasiri
Marco Rojas
Nicholas Stanford

Abstract

The goal of our project was to create a virtual environment where users could synthesize new sounding instruments, learn about music theory concepts such as key signatures, common chords, and how to play music. In our vision, the environment would be highly responsive to the qualities of the sound the custom instrument produces, as well as the user’s musical production. The user could play the instrument manually, using on-screen buttons, or add notes to a “sequencer”, a common interface seen in computer-generated music software. The programming language we have used to create the sounds and play notes is ChucK. ChucK is a strongly-timed language, an eloquent term used to describe the manner in which the virtual machine the language runs on plays sounds precisely when, and for how long, the program specifies. The language makes it easy to generate complex waveforms by stacking various frequencies on top of each other. 3ds Max, a 3d modeling software, was used to create 3d assets for our virtual environment. 3ds Max is user friendly and versatile, making it the ideal software for creating a visually appealing environment. Unity is the software development package that we used to create our interactive environment from the sounds we created in ChucK and the visual components made in 3ds Max. We applied the research we gathered on music theory to create a virtual environment that helps the audience learn more about music.

Introduction

Music and Mathematics, though highly intertwined, are rarely seen as such. Music is expressive, emotional, and tells stories. Mathematics, by comparison, seems logical and objective. Even so, music can be precisely described mathematically, and by observing the mathematics of music, insights can be drawn about the formation of key signatures, mellifluous chords, and the nature of the sound waves we perceive when we listen to an instrument. We hope this musical environment facilitates this type of insight. People can find new ways to think about music, and express themselves through the combination of art and science that is music.

We identified some characteristics of music and sound that can be reasoned about mathematically, and found ways to connect them to visual objects in the environment. We chose relationships that seemed intuitive. An example is the timbre of a sound (the “color” that is a result of the physical characteristics of an instrument or vibrating sound-producer) and the color or texture of an object. Another example is the frequency (or pitch) of the sound produced and the position of an object.

Theory Background(Include relative background and theory pertaining to the subject)

Digital music and audio has become more available to audiences in the last 20 years with the advance of the internet. We researched music theory in order to attempt to make software that could familiarize the audience with the concepts we encountered. The driving idea was to help users develop an appreciation of the distinction between music and sound. We want users to understand that when they hear a note, they are actually hearing a combination of frequencies, the superposition of which creates a precisely defined complex waveform. The user would learn these things all while engaging in an engaging interactive experience that would allow them to construct compositions with multiple parts playing simultaneously. We use the term, “compositions,” very broadly here, as users would be able to make melodies and rhythms, but should feel equally encouraged to simply make interesting sounds. In our software, they would do this using an interface that lets them define a frequency as the base and stack other frequencies, called overtones, on top of the base.

Methodology(complete description of all the work that has been completed)

Every week we did research and built components we believed needed to be added to our project as a whole. We asked ourselves ‘what is the difference between what we already know, and what we still needed to venture out and find?’ Then we searched for those components and found a way to implement them into our project as a whole. As computer science students, there was much for us to learn in the area of music theory. In one project we researched the creation of major scales, a fundamental concept in music, and applied visual elements to our project to help users play notes within a major scale.

The scope of our ambitious ideas required that our project use many different technologies. The software we used to produce computer generated audio was a programming language called “ChucK” that has unique characteristics that make it ideal for sound synthesis. The application used to model our environment, 3ds Max, contains no scripting, but it played a key part in our final project. We built our final project in Unity, a “game engine,” which we most readily adopted the use of because it was most directly related to our previous studies. A capstone project, for some, is a chance to show off what has been previously learned. On the contrary, nearly every aspect of our project required intense research and self-teaching, but we’ve persevered and built a final product to be proud of.

Results/Analysis (the results obtained from our methodology)

An important lesson we’ve taken from this experience is that the amount of forward planning, communication, and work required to accomplish a given objective is difficult, but these are certainly skills that can be improved. One primary reason for this is that unforeseen obstacles and complexities that demand a complete abandonment of previously completed work. Our initial natural model for a synthesizer was computationally prohibitive, and when it was deemed unusable, scrapping a large amount of code took a fair amount of humility. At the time, it seemed that time and effort had been wasted, but on the contrary, we’d become familiar enough with the technologies we’d been using to realize the errors in our implementation. In other areas, as well, the tasks which we believed to be the larger hurdles ended up seeming simple, while some simple tasks became painstaking. For instance, learning to code in ChucK sounded ominous, but effectively was accomplished within days. On the other hand, making a sound play when an object is clicked, and then continue playing until the mouse button is released, was a more daunting accomplishment than we had imagined. Nonetheless, our combined efforts resulted in our successful completion of the project. We created a new experience that allows the user to make and experience music in a new and unique way.

References

Atherton, J.; Wang, G. Chunity. “Integrated Audiovisual Programming in Unity.” NIME. 2018, June 3 - 6.

Cook, P. R.; Salazar, S; Wang, G. “Chuck: A Stongly Timed Computer Music Language.” Computer Music Journal. 2015, 39:4, 10-29.

Benham, C. E; Goold, J; Kerr, R; Newton, H. C. Harmonic Vibrations and Vibration Figures. Facsimile Publisher. 2019. 1909. Reprint.

Tymoczko, D. A Geometry of Music: Harmony and Counterpoint in the Extended Common Practice. Oxford University Press. 2011.


About


We are a team of students at University of Colorado - Denver, computer science department. We are working on our senior design project, and we created this website as suggestion of our professor Dr. Gethner that she is supervise our project and giving us all the feedback that is required to make our project gives the best results. This website will have the data of the project and also will allow us to easily upload all the resources.







"...there is always the pleasure of watching to see what form the figure will take, the interest of seeing it grow and develop itself in a series of most perfect and exquisite curves, and, almost before it is finished, the wonder what the next will be like.
As one becomes more familiar with the working of the instrument, there is the pleasure of being able to repeat a figure at will, or to vary it by altering the phase, for this is an endless hobby; no one has yet exhausted all the possible combinations, the complexity and beauty of which increase as one proceeds."

- Herbert C. Newton in Harmonic Vibrations
About

Team


Marco Rojas

Nicholas Stanford

Zaid AL Yasiri

Biography

Read More

Khalid Adkins

Articles


Music Player


Members Assets


IEEE xplore

IEEE Xplore provides full-text access to IEEE transactions, journals, magazines and conference proceedings published since 1988 and all current IEEE Standards.

Learn More

MathSciNet

MathSciNet is the online version of the American Mathematical Society (AMS) publications Mathematical Reviews, and Current Mathematical Publications.

Learn More

ACM Digital Library

The ACM Digital Library is a research, discovery and networking platform containing: The Full-Text Collection of all ACM A collection of curated and hosted full-text publications, etc

Learn More

Auraria Library

This is the link to the Auraria library

Fall Semester Progress


Week 1-5 Information Gathering

    Exploration of music theory concepts.  Topics include:

        the Tonnetz, Euclidean Rhythms, theory of chord progressions, 

        and the mathematics of musical scales.

    Exploration of many examples of music visualization.

    We considered applying natural language analysis (n-gram analysis) to prediction or generation of notes.

    Technologies that can be used to manipulate and generate music and musical sounds            


Week 6 - 9/30 Beginning of ChucK music programming exploration

    Discovered how to manipulate musical notes and their associated frequencies with the MIDI-related functions in ChucK

    Manipulate and control time in ChucK.

    Create sound with unit generators

    change the timbre of unit generators using parameters and 


Week 7-8 Further Chuck exploration

    Notes and chords represented in arrays and used in functions 


Week 9 Chunity Discovery and exploration

    Connect ChucK and Unity and use ChucK to generate sound within Unity using variables from within 

    the Unity environment

    

Week 12

    Further exploration of timbres using unit generators and envelopes and an understanding of how 

    musical instruments make sound.  


Spring Semester Progress


Week 1: 

Nicholas

    - Finish waveform synthesizer

Marco

    - Use variables from chuck sound generation and analysis in Unity

    - game with two connected players

Khalid

    - finish guitar

    - Virtual studio environment

Zaid

    - Work with Python to finish Virtual MIDI pad (Using Gamepy, numpy) and the music library 


Week 3:

Nicholas

    - Make user interface to more easily adjust overtones/vibrato

Marco 

    - explore UAna possibilities

    - 2D or 3D objects changing shape or attributes based on UAna outputs

    - 2 person 

Khalid

    - more insturuments and studio inspiration

Zaid

    - will finish the Python-MIDI-PI project

    - Helping Khalid to create more musical instruments using 3D max


Week 5:

Nicholas

    - Make user-unfriendly synthesizer interface

Marco 

    - 

Khalid

    -     

Zaid

    - Start synthesizer interface

    

Week 6:

Khalid

    - Rudimentary level 

Zaid

    - Finish synthesizer interface

Nicholas

    - Figure out how to eliminate clipping

Marco

    - Sequencer build


Week 9:

Khalid

    - Stuff

Zaid

    - Stuff

Nicholas

    - Combine Synth with Khalid/Marco's interfaces

Marco

    - Stuff

Ethical Considerations of Computing


Ethical Considerations of Computing
One of the goals of the Senior Design program at CU Denver is to encourage students to consider the ethical implications of our contributions to society.  While philosophical questions may not initially seem relevant to data science and software development, in reality they should be central to the decisions we make going forward into our careers due to the profound impact software has on modern life.

Presentations


About this page

What did we do in - class

In this page we will share all the Presentations that we had in class. we will also include some of the notes that we have taken to improve our results in order to build and present a perfect final project

Read More

In - class Presentations

Please upload all the Word and PowerPoint files that you have created for Senior Design I

Music and colors.pptxUnderstanding Unity.pptxSenior_Design_I.pdf

Read More

Contact


  • Denver, Colorado, United States