Processing can be loaded lots of external data, among which there are three types very commonly used. They are image, audio and video separately. In this chapter, we are going to talk about how to load audio and video in detail, combining with events. In the end, you can create your own music keyboard or music palette.  

Read Image  

Before we start, let’s look back on the method of image loading.  

Image Related Functions  

Before we use these functions, we need to create an image object through PImage. Then we can use these functions to define all kinds of image properties. Don’t forget to store your image sources into data file before running your program.  

Music Loading, Play & Stop  

In the following, we begin to formally introduce the invocation of music to you. Very similar to image loading, you have to declare an audio object at the beginning. You can refer to the example below fro the actual grammar. Code Example (10-1):

  [cceN_cpp theme=”dawn”] import processing.sound.*; SoundFile sound; void setup() { size(640, 360); background(255); sound = new SoundFile(this, “1.mp3”); } void draw() { } void keyPressed() { //Play sound if (key == ‘p’) { sound.play(); } //Stop sound if (key == ‘s’) { sound.stop(); } } [/cceN_cpp]  

Preparation: Processing itself do not carry any sound library. You need to download it by yourself. So, before writing your code, you’d better make the following preparations. Add library to Processing. Here’s the common practice. Choose from the menu bar for “Tool”– “Add Tool”, and then shift to “Libraries”. Input the key words of library into the search column so that you can download and install it directly.  

However, if we use this function in our domestic country (in China), we cannot download it by connecting the web directly. We have to startup VPN. Even though we start it, there will be instable conditions. So you have to be patient to try it several times. This is the most convenient method of loading. If you cannot install, you have to download from the official website by manual. (https://processing.org/reference/libraries/) Because the manual installation method is very complicated, we will further discuss it in the other chapter.   Code Exaplain: The sound library can work properly after preparation completed. Cope the above code, click RUN, then it will operate. Press the key “P” to play music, “S” to stop music. If it is used to the program, we need to load it first. At the beginning, we have to add a sentence “import processing.sound.*”. “import” is the key word, meaning loading literally. Add the library name behind “import”, then it will load the library. The tail usually follows a mark of “*”, thus it will load all of the library related classes into the program without having to add them one by one manually. In the second sentence, ” SoundFile sound; ” has declared an audio object. SoundFile is similar to PImage. Within function setup, ” sound = new SoundFile(this, “1.mp3″); ” is used to create an object and define its read path. Here we actually have already started to use a new concept Class. Right now we do not discuss it deeply. We only need to know it is a fixed writing method and the last parameter is for filling the address of music source. Among keyPressed() events, ” sound.play() ” and ” sound.stop() ” relatively work as the effect of play and stop. “.” in the middle indicates a member function that play and stop belong to audio objects. We can regard member function as the function included in the object. It is belong to this object, which is defined beforehand. Later, when we need to play multiple audio objects, we only have to add “.play()” behind the relative variable name. Audio sources shall be stored into data file under the same catalogue of sketchfile(with pde suffix). If there is none, you can manually create one. Don’t forget to write function draw. Although you didn’t draw any graphics, it is necessary to play music successfully. The above procedure seems quite complex, but you need to add several sentence of code only, then you can realize play function. It is very convenient. Processing supports common audio formats like mp3, wav, ogg, etc..  

Music Speed Control

  The following examples will start to become much interesting. Processing has provided some functions that can control music playing speed. At the same time, tones will change with music playing speed. When we use mouse to control, it will generate very psychedelic effect. Video website: http://v.qq.com/x/page/e0322yw5el0.html   Code Example (10-2):  

[cceN_cpp theme=”dawn”] import processing.sound.*; SoundFile sound; void setup() { size(640, 360); background(255); sound = new SoundFile(this, “1.mp3”); } void draw() { float speed = mouseX/(float)width * 3; sound.rate(speed); float vol = mouseY/(float)height * 4; sound.amp(vol); } void keyPressed() { //Play sound if (key == ‘p’) { sound.play(); } //Stop sound if (key == ‘s’) { sound.stop(); } } [/cceN_cpp]  

Code Explain: Function .rate() controls the audio playing speed. The value in the parenthesis determines the fast and slow of playing speed. When the value is 1, the playing speed is normal. When it is beyond 1, then accelerate; while it under 1, then decelerate. Function .amp() controls the audio volume. The value in the parenthesis determines the volume value. When it is 1, the volume value is normal. When it is beyond 1, then increase volume; while it is under 1, then decrease volume. Here we have built two local variables speed and vol as parameters to be loaded into. Therefore the horizontal coordinate of mouse will change the music tone, and the vertical coordinate will change the music volume.  

Video Play And Stop  

In Processing, video loading is similar to audio loading. You have to download video library first. (https://processing.org/reference/libraries/video/index.html )   Code Example (10-3):  

[cceN_cpp theme=”dawn”] import processing.video.*; Movie mov; void setup() { size(640, 360); background(0); mov = new Movie(this, “1.mov”); } void movieEvent(Movie movie) { mov.read(); } void draw() { image(mov, 0, 0,640,360); } void keyPressed() { if (key == ‘p’) { mov.play(); } if (key == ‘s’) { mov.stop(); } if (key == ‘d’) { mov.pause(); } } [/cceN_cpp]  

Video Screenshot:  

Code Explain: The first sentence ” import processing.video.*; ” is used to load video library. The second sentence ” Movie mov; ” is used to declare video object. Among it, the function of ” Movie ” is similar to PImage. In function setup, the effect of ” mov = new Movie(this, “1.mov”); ” is to create object and define its read path. The last parameter shall be filled in with the address of video source. Behine setup, movieEvent represents video event. It is used to update and read video information. “mov.read()” in the event means read. Except for displaying images, function image can display video too. We can regard video object as a dynamic picture. The first parameter, we fill in the variable name of the video object. The second and third parameters are the horizontal and vertical coordinate drawn by video. The fourth and fifith parameters decide the length and width of video display. Function .play() means play. Function .stop() means stop, and it will reset the video. Function .pause() means pause. It will interrupt the current playing, which will continue until function .play() is invoked.

  Video Speed Control  

Code Example (10-4):  

[cceN_cpp theme=”dawn”] import processing.video.*; Movie mov; void setup() { size(640, 360); background(0); mov = new Movie(this, “transit.mov”); } void movieEvent(Movie movie) { mov.read(); } void draw() { image(mov, 0, 0, width, height); float newSpeed = mouseX/(float)width * 4; mov.speed(newSpeed); } void keyPressed() { if (key == ‘p’) { mov.play(); } if (key == ‘s’) { mov.stop(); } if (key == ‘d’) { mov.pause(); } } [/cceN_cpp]    

Code Explain: Function .speed() can be used to control the video playing speed. When the parameter value is 1, the playing speed is normal. When the value beyond 1, then accelerate; while it under 1, then deccelerate. Because we have built local variable newSpeed and imported it into function setSpeed(), the coordinate of mouse will influence the video playing speed directly. As for more examples about video, you can refer to Libraries – Video in the case library.  

Processing Common Events

Previously, we have introduced keyPressed() event only. It will be triggered after the keyboard pressed. In the following, we will introduce other common events in Processing for you.  

The usages of the above events are similar to keyPressed. They have no sequence in code writing. In other words, no matter which event you placed before or behind the function setup, you get the same result. The execution order is only related to the triggering condition of the event itself. Only if the condition is met, then it will execute. Events above are all very easy to understand. You just have to do a small experiment, then you can quickly grasp their usages.  

Event Flow  

We can use an example to know the execution order of events.   Code Example(10-5):  

[cceN_cpp theme=”dawn”] void setup() { frameRate(2); println(1); } void draw() { println(2); } void mousePressed() { println(3); } void mouseMoved() { println(4); } void mouseReleased() { println(5); } void keyPressed() { println(6); } void keyReleased() { println(7); } [/cceN_cpp]  

Code Explain: In function setup, function frameRate() has set the operate speed rate of the program to be 2 frames per second. Lowering the frame rate can help us observe the output in the console in case that events triggered will immediately be brushed by new data to the back. Try to move your mouse, click mouse, release mouse and observe the output result. Get to know the event execution order through println. What is worthwhile to pay attention is drawing functions can not be written into other events except function draw, or it can not display. If we want to control the hide and display of graphic components through events like keyPressed , we may consider to build bool variable as medium. Events will execute in order. Only after all of the code in current event are implemented, it will execute the code in next event.  

Comprehensive Example–Music Keyboard  

Combining with the newly grasped events, we can add new interactions to our program. Next, only with few minutes, we can analog a music keyboard easily. Video website: http://v.qq.com/page/q/o/t/q0322w02zot.html   Code Example (10-6):  

[cceN_cpp theme=”dawn”] import processing.sound.*; SoundFile sound1, sound2, sound3, sound4, sound5; boolean key1, key2, key3, key4, key5; void setup() { size(640, 360); background(255); noStroke(); sound1 = new SoundFile(this, “do.wav”); sound2 = new SoundFile(this, “re.wav”); sound3 = new SoundFile(this, “mi.wav”); sound4 = new SoundFile(this, “fa.wav”); sound5 = new SoundFile(this, “so.wav”); } void draw() { background(255, 214, 79); rectMode(CENTER); float w = width * 0.1; float h = height * 0.8; if (key1) { fill(255); } else { fill(238, 145, 117); } rect(width/6, height/2, w, h); if (key2) { fill(255); } else { fill(246, 96, 100); } rect(width/6 * 2, height/2, w, h); if (key3) { fill(255); } else { fill(214, 86, 113); } rect(width/6 * 3, height/2, w, h); if (key4) { fill(255); } else { fill(124, 60, 131); } rect(width/6 * 4, height/2, w, h); if (key5) { fill(255); } else { fill(107, 27, 157); } rect(width/6 * 5, height/2, w, h); } void keyPressed() { if (key == ‘a’) { sound1.play(); key1 = true; } if (key == ‘s’) { sound2.play(); key2 = true; } if (key == ‘d’) { sound3.play(); key3 = true; } if (key == ‘f’) { sound4.play(); key4 = true; } if (key == ‘g’) { sound5.play(); key5 = true; } } void keyReleased() { if (key == ‘a’) { key1 = false; } if (key == ‘s’) { key2 = false; } if (key == ‘d’) { key3 = false; } if (key == ‘f’) { key4 = false; } if (key == ‘g’) { key5 = false; } } [/cceN_cpp]  

Code Explain: We need to create multiple audio objects to read the relative sound information so as to play different sounds when different keys triggered. Here we use a new event keyReleased(). The function of this event is to restore the keyboard color to its original color. When release the key, it will be triggered. The 5 boolean values declared in the head are used to detect the status of key.  

Comprehensive Example–Music Palette 1  

Besides keyboard event, mouse event is a good thing that we have to use it flexibly. The following example is for us to create a music palette, among which we have used two mouse related events. Video website: http://v.qq.com/x/page/d03224cwykv.html   Code Example(10-7):  

[cceN_cpp theme=”dawn”] import processing.sound.*; SoundFile sound1, sound2, sound3, sound4, sound5; boolean isDragging; void setup() { size(640, 360); background(255, 214, 79); noStroke(); sound1 = new SoundFile(this, “do.wav”); sound2 = new SoundFile(this, “re.wav”); sound3 = new SoundFile(this, “mi.wav”); sound4 = new SoundFile(this, “fa.wav”); sound5 = new SoundFile(this, “so.wav”); } void draw() { if (isDragging) { fill(107, 27, 157, 100); ellipse(mouseX, mouseY, 16, 16); } } void mouseDragged() { isDragging = true; if (mouseX > 100 && mouseX < 105) { sound1.play(); } if (mouseX > 200 && mouseX < 205) { sound2.play(); } if (mouseX > 300 && mouseX < 305) { sound3.play(); } if (mouseX > 400 && mouseX < 405) { sound4.play(); } if (mouseX > 500 && mouseX < 505) { sound5.play(); } } void mouseReleased() { isDragging = false; } [/cceN_cpp]  

Code Explain: We hope that only when we have pressed the mouse and dragged it, we can draw pictures. So we need to build a boolean variable isDragging to obtain current status. When dragging the mouse, isDragging becomes true value so that drawing functions within Draw will be executed. It will leave traces on the screen. When we release the mouse, isDragging becomes false value. So the drawing functions in function draw will stop execution. We have designed several triggering conditions in mouse dragging event. For example, when the horizontal coordinate of mouse is among 100 and 105 pixels, music will be played automatically. This makes the screen created several invisible strings. Only if the mouse passes through the certain areas, it will trigger the relative music.

Comprehensive Example–Music Palette 2( Updated Version)   The effect of the above example is already good enough. But if we watch it carefully, we will find lots of problems. For example, when the mouse moves very fast, it will leave a round point on the screen every time it moves. It is not a coherent straight line. Meanwhile, it also causes some music leak. While when the mouse moves very slowly, passing through the position when the horizontal coordinate is among 100 and 105, it will broadcast music for multiple times within a very short time, which gives you a feeling of being stucked . All of these problems, we can solve it through the example below. You can watch videos in the link below: http://v.qq.com/x/page/w03226o4y4l.html   Code Example(10-8):  

[cceN_cpp theme=”dawn”] import processing.sound.*; SoundFile sound1, sound2, sound3, sound4, sound5; boolean isDragging; void setup() { size(640, 360); background(255, 214, 79); noStroke(); sound1 = new SoundFile(this, “do.wav”); sound2 = new SoundFile(this, “re.wav”); sound3 = new SoundFile(this, “mi.wav”); sound4 = new SoundFile(this, “fa.wav”); sound5 = new SoundFile(this, “so.wav”); } void draw() { if (isDragging) { stroke(107, 27, 157, 100); strokeWeight(10); line(mouseX, mouseY, pmouseX,pmouseY); } } void mouseDragged() { isDragging = true; if ((mouseX – 100) * (pmouseX – 100) < 0) { sound1.play(); } if ((mouseX – 200) * (pmouseX – 200) < 0) { sound2.play(); } if ((mouseX – 300) * (pmouseX – 300) < 0) { sound3.play(); } if ((mouseX – 400) * (pmouseX – 400) < 0) { sound4.play(); } if ((mouseX – 500) * (pmouseX – 500) < 0) { sound5.play(); } } void mouseReleased() { isDragging = false; } [/cceN_cpp]  

Code Explain: Here we have used two variables pmouseX and pmouseY carried in Processing system itself. They are similar to mouseX and mouseY but what they obtained are the coordinate of mouse in the last frame. In Function draw, we have used function line() to replace the original function ellipse(). This makes the coordinate of last frame connected with the coordinate of current frame directly. So we can draw coherent straight lines or curves. In event mouseDragged, we have designed a new triggering condition. Through judging whether the coordinate of the last frame and current frame are in the same side to know whether a certain coordinate is crossed. Take this condition as an example: ” if ((mouseX – 100) * (pmouseX – 100) < 0) “. Among it, from the positive and negative value resulted from ” mouseX – 100 “, we can know whether mouseX is at the right or left of the horizontal coodinate 100. Similarly to “pmouseX – 100”. Therefore, when the two points in the front and back are not in the same side, a positive multiplies a negative, it will get a new negative number. Thus the execution condition is satisfied. The above is a simplified expression, which has cleverly used a certain mathematical algorithm–Two negatives multiply will create a positive. You can also divide it into two situations to discuss separately. However, it is much more complicated to write judgement conditions. The judgement conditions “if ((mouseX < 100 && pmouseX >= 100) || (mouseX > 100 && pmouseX <= 100))” is equivalent to the determining conditions of the source code.  

Relative Functions About Audio&Video Control  

The above-mentioned functions are sufficient for general usage scenarios. If you want to dig it deeply, here I have collected some common functions relative to audio and video for you. You can explore its usages according to your own requirements. 

For more introduction, you can refer to documents from official website. Audio (https://processing.org/reference/libraries/sound/index.html) Video (https://processing.org/reference/libraries/video/index.html)

This article comes from designer Wenzy.  

Relative Readings:  

Interesting Programming Guidance for Designer——Processing Initial Touch Interesting Programming Guidance for Designer–Create Your First Processing Program Interesting Programming Guidance for Designer–Get Your Picture Running(Part One) Interesting Programming Guidance for Designer–Get Your Picture Running(Part Two) Interesting Programing Guidance for Designer–Program Process Control- Loop Statement Interesting Programming Guidance for Designer–Program Process Control–Condition Statement (Part One) Interesting Programming Guidance for Designer–Program Process Control–Condition Statement (Part Two) Interesting Programming Guidance for Designer–Custom Functions and Fractal Recursion Interesting Programming Guidance for Designer–Custom Functions and Fractal Recursion Interesting Processing Programming Guidance for Designer–Color Control