ACE Studio is every producer’s dream. Imagine typing lyrics and having them sung to the melody you play or draw on a simple grid-based interface. It lets anyone generate realistic vocals, perfect for creating and demonstrating toplines, adding backing vocals, choir parts, and more. With a variety of singers and languages to choose from, ACE Studio showcases the magic of combining the latest AI technology with a workflow designed for producers and composers.
Creating such an advanced tool might seem like it requires massive departments, extensive funding, and the involvement of AI giants like Google or OpenAI. Surprisingly, ACE Studio began as an experiment between a couple of friends.
In this series, we unveil the creation stories behind today’s cutting-edge music technology. From apps that sing to producers crafting tomorrow’s hits, we highlight the people and stories behind the products.
Finding His Calling: Sean Zhao’s Journey
When Sean Zhao was a kid, there were no musical instruments at home. However, there were still things that made noise. He discovered he could play melodies on the family’s touchtone phone, which produced different notes as he pressed the keypad numbers. Fascinated, Sean transcribed these melodies into number sequences. One day, he heard a song that had a melody similar to one he’d “composed” earlier. “I felt incredibly happy when I heard it; this experience planted the seed of my love for music”, Sean recalls.
Sean later took piano lessons in middle school, but it was when he was gifted a simple electronic dictionary with a built-in code compiler and a few games that his curiosity with coding was ignited. “I wrote my first lines of code on that little device; a simple for-loop that printed numbers from 1 to 100,” he says.
Sean’s interest in technology grew as he did. “I chose software engineering as my major in university and obtained a master’s degree in computer game engineering. My first job was as a game developer, creating mobile games and working on game engine development at Tencent.”
Sean’s love for music was ever-present, and in early 2016, he saw the musical potential in early neural networks. Neural Nets, the foundation of today’s AI revolution, can learn to recognize patterns and make decisions without needing explicit programming.
“In late 2016, an AI named DeepBach emerged, capable of generating music in Bach’s style,” Sean explains. “I was shocked by this technology; it reignited my childhood dream that I had once thought unattainable. I realized that perhaps I could achieve my dream through a different path—by enabling anyone, regardless of skill level, to create their own music.”
Sean was fortunate to meet like-minded dreamers, Joe and Conger. “Joe has deep insights, and Conger is a musician with high artistic taste.” Together, they formed a company and started building their dream.
The idea was to make music creation accessible to everyone. Non-professionals often feel embarrassed to ask singers to perform their songs or struggle with endless revisions. We thought that singing synthesis would help lower these barriers, helping everyone to create their own music.
The revolution of AI made this once near-impossible task seem somewhat achievable. “We started with a simple concatenative singing synthesis engine. Although the results were not initially great, users still created impressive works. That was when we realized people needed this product.”
Try, try… and try again!
Together, they doubled down, worked hard, and ACE Studio started to take shape. For Sean, Joe, and Conger, the main goal of ACE Studio is to give producers a way to express their artistic vision and experiment with their style. “We’d love it if one of our customers has a hit song that was made possible by ACE Studio.”
Of course, AI is a hot topic among today’s creative audiences, and there’s growing concern around the impact it could have on the creative arts and industries. Sean acknowledges: “Thanks to the development of large language models (LLMs) and many pre-trained self-supervised models, generative models in other modalities, especially video and audio, have been developing very rapidly. These models have significantly lowered the barriers to content creation, and we can expect to see many unexpected and innovative works emerge. However, as AI-generated content becomes increasingly indistinguishable from real content, there is a growing need to further refine laws and regulations to better govern AI-generated content and to protect human copyrights and other rights.”
Many creatives naturally have concerns about the development and pace of AI, and what it may mean for creative industries. The ACE Studio team is positive:
“AI music is developing rapidly, and there are already powerful AI systems like Suno and Udio that can generate high-quality music on demand. However, we believe that musicians and composers will never be completely replaced. Music creators need to express their emotions and inspirations, and directly generating music with AI or describing it in words can never precisely produce the results they envision.
We should expect that much commercial music, such as music for advertisements or background music for bars and restaurants, could be generated by AI. However, music for enjoyment needs a human touch to infuse it with soul, and often it is more than just the audio waveform itself. AI can only learn patterns from existing songs; it cannot create new genres, and there is no human soul or story to listen to as part of the music it creates. For example, if AI music had existed before the advent of jazz, AI would never have been able to emulate jazz.”
ACE Studio represents an incredibly useful, creative application of AI – it powers an indispensable tool for composers and producers, helping them achieve their own artistic vision. So what’s next for this incredible app?
“Currently, we have integrated many AI tools into ACE Studio, such as converting vocals to MIDI and lyrics, and extracting vocals, accompaniment, and various instruments from audio. In the future, we will introduce more AI tools, such as suggesting accompaniment based on vocals and making it easier for users to generate choirs. Additionally, we plan to gradually add more built-in vocal effects like reverb, equalizer, and compression, and we are also considering supporting third-party plugins.”
We can’t wait to see it!
Download ACE Studio from Muse Hub today, and experience the power of AI vocalists.
