Block, under Jack Dorsey, has released a new voice model from the company's research laboratory. This model can run locally and incorporates expressions such as laughter, pauses, and crying, sounding incredibly realistic like a human. The company claims that this model, named Dessa, is trained entirely on human voices and can operate with just one NVIDIA GPU.
According to Block, Dessa is a full-duplex model, which means it can speak and listen simultaneously, just like the GPT-Live model released by OpenAI earlier this year. Typically, voice models are used in conjunction with large language models for knowledge retrieval. Block indicates that the knowledge of Dessa comes from the dialogue data it has been trained on.
Block indicates that the current model appears unnatural and even irritating because the process of generating speech includes text. Many models, such as ElevenLabs, v4, or Fish Audio Drama 3, use speech tags to drive the emotions in speech generation. The company behind Dorsey claims that emotions are built into their model. However, it is not yet clear whether these emotions are guided by some kind of tag.
"Interacting with the current voice AI is simply terrible. It converts your voice into text, losing a great deal of information that text itself cannot convey. Even the world's most powerful AI systems are unable to detect your emotions. We find this absurd and believe it leads to a worse user experience, and may even be dangerous," the company stated in a blog post.
Company researchers have released multiple dialogue examples of Dessa, in which this intelligent entity speaks with different voices, speaking speeds, and emotions. For instance, in one dialogue, the robot mentions that it is tired from work, and at that time, it indeed sounds very tired.
In the following example, when the robot talks about the person who makes it feel comfortable, its voice carries emotion, as if it’s on the verge of breaking down.
Another example shows that members of the Block team are playing games with this voice assistant.
Block indicates that this model can run locally on consumer-grade computers, such as those with MacBook M4 Max or higher configurations. However, at present, Dessa has not yet been made available to the public, while Block has invited security researchers to test this model. This is likely to be in order to examine how voice agents might deceive people on the other end.
Block specializes in financial services and sells its solutions to small businesses. Square has already been using AI to handle ordering operations. However, with its own voice model that can run on devices, it is able to support more enterprise and consumer experiences at a lower cost.
