tekBlueprint

Continuing with the topic of using AI, in the previous article “Not all AIs are equally useful?” I didn’t mention music generation, although I’ve been following the progress in this direction for some time.

There will be no technical details, but rather a general impression of the current state of the art in music generation.

Preface

Interest in music generation arose when it became necessary to create tracks for one of the projects. Of course, it would have been possible to use ready-made tracks. There are quite a lot of different sound banks, with a huge selection, naturally not always free for commercial use :). In addition, I still wanted to do something new and unique, and not copy public works.

The second option is to do it myself. The main type of music required was ambient with absent or minimal vocals, which does not require complex instruments. Although this is contradicted by the orchestral soundtrack to Dune 2021. And in general, it was possible to do it yourself, there is Arturia MiniLab 3 that I once bought, and Analog Lab gives a large number of ready-made instruments and samples for use. But again, everything depends on time constraints, although it would be quite interesting to try.

For these reasons, I began to look towards generating music using neural networks.

What was the problem

During the previous year, the quality of the generated music left much to be desired. It didn’t matter whether they were open or closed models. It was very noticeable that they couldn’t create complex and interesting melodies, as well as structured compositions. The voices in them were even worse – singing robots :).

At first, there were some attempts to generate music based on diffuse models. To be honest, I don’t even remember what I used then. I think it was Riffusion for AUTOMATIC1111 / Stable Diffusion web UI, with a time limit and low quality.

Cloud services SUNO AI, Udio did not produce much better results. A little later, I looked at another service Mubert, which produced better results. But it was still noticeable that these were generated tracks.

How it is now

I judge by services like SUNO AI, Udio. The quality has become much better. There are tracks where it is already difficult to distinguish from the real ones, if you do not listen closely. There are errors, but in general it is acceptable for home use (or something like that). For example, the previously free Riffusion has clearly become better, an example will be below in the article. Naturally, professional musicians will reject this right away :)

Some models even make trial steps towards dividing the generated tracks into tracks (stem), but so far not very successfully.

The situation with open models is still sad. ACE-Step was recently made publicly available. Available for ComfyUI, if you don’t want to mess around with Python. The generated tracks have a very noticeable synthetic voice and extraneous noise, although the overall composition can be interesting.

A small digression, with voice generation the situation is a little better. For example, with Sesame AI (Conversational Speech Model) it can already create quite decent voice messages with intonation, emotions and can be fine-tuned on any specific voice. Available for local use.

Example (Programmers’ Symphony)

The lyrics to a song about developers, unexpectedly, quickly generated through LLM and made some minor edits

Programmers’ Symphony

[verse]
In the glow of the screen, we weave our designs,
Python whispers, Java rains in midnight lines.
Silent solvers of mysteries unseen,
In code, we find where magic has been sown.

[chorus]
Programmers, in the code we find our might,
Turning ones and zeros into digital light.
Endless loops but we never tire,
Building castles where imagination fires.

[verse]
Fingers dance fast on keys so cold,
Midnight approaches, yet our resolve is bold.
Through each line, a story unfolds,
In the binary night, we stand tall and bold.

[chorus]
Programmers, in the code we find our might,
Turning ones and zeros into digital light.
Endless loops but we never tire,
Building castles where imagination fires.

[verse]
Debuggers search through lines so thin,
A single bug can make us begin again.
But when it clicks, oh what joy it brings,
In the silence, our triumph sings.

[bridge]
When systems crash and chaos looms,
We rise from ashes, rebuild with room.
No mountain too high, no valley too deep,
In every byte, our spirit we keep.

[chorus]
Programmers, in the code we find our might,
Turning ones and zeros into digital light.
Endless loops but we never tire,
Building castles where imagination fires.

[verse]
From nothingness to something grand,
Our digital footprint spans the land.
In lines of code, our legacy grows,
Programmers' symphony, forever flows.

The results themselves. The first option is generation through an open local model, and the second through a commercial one:

ACE-Step

Riffusion

As I said before, the local one is not very good, maybe it will be better in the next versions. It can even sing in different languages, but with a terrible accent.

But the commercial solution really works quite well. For example, ByteBeat. Moreover, I used the free version with the FUZZ 1.0 model. Perhaps the paid version with FUZZ 1.0 Pro will be even better.

Conclusion

I caught myself thinking that perhaps from the musicians’ point of view, it looks the same as when I look at non-programmers who have written something using neural networks and are happy :) Most likely, there will be a lot of errors and shortcomings, but for me personally, these examples from Riffusion sound quite decent. Considering the lack of musical education, the ability to make a track for your needs is already worth a lot.

But the initially needed tracks in the ambient style, most likely, will have to try to make yourself. I’m not sure that it will be possible to explain to the neural network what exactly is needed and I want to try to do it myself. It’s also interesting, in the future there will be separate designations for hand-made things - “craft” music, “craft” images, “craft” code of two years of aging with tech debt :)