Simulating A Living Mona Lisa

[WhistlingJack]

Mona Lisa "speaks" thanks to Japanese scientist

Wed May 31, 2006 8:51 AM BST

By Toshi Maeda

TOKYO (Reuters) - The Mona Lisa's smile may always remain a mystery, but it is now possible to hear what her voice would have sounded like, thanks to a Japanese acoustics expert.

Dr Matsumi Suzuki, who generally uses his skills to help with criminal investigations, measured the face and hands of Leonardo da Vinci's famous 16th century portrait to estimate her height at 168 cm and create a model of her skull.

"Once we have that, we can create a voice very similar to that of the person concerned," Suzuki told Reuters in an interview at his Tokyo office last week. "We have recreated the voices of a lot of famous people that were very close to the real thing and have been used in film dubbing."

The chart of any individual's voice, known as a voice print, is unique to that person and Suzuki says he believes he has achieved 90 percent accuracy in recreating the quality of the enigmatic woman's speaking tone.

"I am the Mona Lisa. My true identity is shrouded in mystery," the portrait proclaims on a Web site at promotion.msn.co.jp/davinci/voice.

"In Mona Lisa's case, the lower part of her face is quite wide and her chin is pointed," Suzuki explained. "The extra volume means a relatively low voice, while the pointed chin adds mid-pitch tones," he added.

The scientists brought in an Italian woman to add the necessary intonation to the voice.

"We then had to think about what to have her say," Suzuki said. "We tried having her speak Japanese, but it didn't suit her image."

Experts disagree over who was represented in the portrait, with some saying the smiling woman is Leonardo himself, or his mother.

The team also attempted to recreate Leonardo's own voice in a project timed to coincide with the release of the film "The Da Vinci Code." Suzuki said he was less confident about its accuracy because he had to work from self-portraits where the artist wore a beard, concealing the shape of his face.

Suzuki's work has made contributions to criminal investigations -- in one case after he successfully aged a person's voice by a decade. A recording of the voice was broadcast on television, leading to the apprehension of a suspect.

© Reuters 2006.

 

[GNC]

Does she say, "Hello tiger!"?

 

[Cavynaut]

Or,

"I'm in a book by that Dan Brown y'know".

 

[Graylien]

I suspect that this is absolute nonsense. Firstly, vocal range is determined by the structure of the larynx (which you can't deduce from a painting) - not the structure of the face. (The reason Castrati sing at a high pitch is because a side effect of removing the testes is that their larynx ceases to develop.)

And secondly, a person's speaking voice isn't just dependant on their physique. It develops - at a very young age - from imitating the people around us. We don't just imitate their accents or patterns of speech - we also imitate their speaking 'pitch'.

This is why people in amateur choirs often end up in the wrong sections. For instance, women assume that if they have a high-pitched speaking voice, they must therefore be a soprano. It ain't necessarily so. They may well be an alto who happened to grow up imitating a family member who spoke with a high-pitched voice.

 

[EnolaGaia]

Here's an AI simulation of La Gioconda conversing. It's one of multiple "Living Portraits" generated by the R&D team cited in the article below. If you go to the linked article and click on the YouTube link you'll get an introduction to the procedures used, then a series of similar living portrait simulations (3 per subject).

The most remarkable thing about the generated living portraits is that they visibly vary in facial expressiveness and expressive stylings, and these variations are directly attributable to the (contemporary) models whose facial maps were employed to dynamically manipulate the painting's features.

I'm not sure why this is characterized as 'terrifying'. Striking? Certainly!

This Animated Mona Lisa Was Created by AI, and It Is Terrifying

The enigmatic, painted smile of the "Mona Lisa" is known around the world, but that famous face recently displayed a startling new range of expressions, courtesy of artificial intelligence (AI).

In a video shared to YouTube on May 21, three video clips show disconcerting examples of the Mona Lisa as she moves her lips and turns her head. She was created by a convolutional neural network — a type of AI that processes information much as a human brain does, to analyze and process images.

Researchers trained the algorithm to understand facial features' general shapes and how they behave relative to each other, and then to apply that information to still images. The result was a realistic video sequence of new facial expressions from a single frame. [Can Machines Be Creative? Meet 9 AI 'Artists']

For the Mona Lisa videos, the AI "learned" facial movement from datasets of three human subjects, producing three very different animations. While each of the three clips was still recognizable as the Mona Lisa, variations in the training models' looks and behavior lent distinct "personalities" to the "living portraits," Egor Zakharov, an engineer with the Skolkovo Institute of Science and Technology, and the Samsung AI Center (both located in Moscow), explained in the video. ...

FULL STORY: https://www.livescience.com/65573-mona-lisa-deepfakes.html

 

[ChasFink]

I'm sure "terrifying' is one of those words that's been proven to work as clickbait.

As for the Mona Lisa simulations, I get the distinct impression that a lot of the "look" of them wound up being part of the actual looks of the training models, not just their "personalities" - the way certain muscles move being related to the actual shape of the model's features. The one in the middle looks very lifelike, but also really doesn't look a lot like La Gioconda. I didn't bother watching the video, so I don't know if this was addressed.

 

[EnolaGaia]

...As for the Mona Lisa simulations, I get the distinct impression that a lot of the "look" of them wound up being part of the actual looks of the training models, not just their "personalities" - the way certain muscles move being related to the actual shape of the model's features. ...

Agreed ... As I understand it, the Mona Lisa features were rendered to provide a sort of graphic template onto / through which the models' captured dynamic parameters (e.g., movements; shifts in features' relative locations and sizes) were projected.

The models' own facial configurations were the referential standard for plotting such relational shifts (e.g., how much wider eyes opened; how the corners of the mouth moved). Plotting changes from the models' original coordinate mapping onto the Mona Lisa template had the effect of animating the target facial map (Mona Lisa) with respect to changes from a different coordinate mapping - i.e., changes whose magnitude and orientation didn't necessarily reflect the (e.g.) musculature portrayed in the Mona Lisa template.

As a result, the dynamic 'skeleton' of the models' maps showed through the template's 'skin', as if the models were wearing a close-fitting elastic Mona List mask.

If nothing else, the viewer's ability to see the subtle variations caused by this translation is an interesting illustration of how fine-grained our facial recognition and visual tracking abilities are.

 

[blessmycottonsocks]

Does she say, "Hello tiger!"?

"If I keep smiling enigmatically like this, he's promised to take me for a spin in his helicopter..."

 

[ChasFink]

"If I keep smiling enigmatically like this, he's promised to take me for a spin in his helicopter..."

I'd hold out for the ornithopter myself.

 

[Swifty]

Does she say, "Hello tiger!"?

.. or "Is this going to take much longer?"

 

[Mikefule]

The irony of "scientists" setting out to do something that is inherently not susceptible to proof or disproof.

Whether they say "she sounded like this" or "she probably sounded like this", we can never know if they are right.

She may have had a long or short tongue, a lisp, an affectedly high pitched or low pitched voice, an accent linked to her region or social group. And the painting — which I have seen and acknowledge to be a fantastic work of art — may not be accurate proportionately.

I am strangely reminded of those reconstructions in the news media: the child who has been missing for 10 years may look like this now — assuming that she has this hairstyle and colour, has been exposed to this amount of sun, has had roughly this balance of exercise -vs- diet, has not been scarred by acne, or in an accident, and has not developed a visual impairment that causes her to squint, or to need spectacles... OK, we made it up.

 

[Frideswide]

where would we be if we only ever explored things that we knew in advance would give a firm answer? What happened to "for the joy of it" as a motivation to explore stuff?

 

[Mikefule]

where would we be if we only ever explored things that we knew in advance would give a firm answer? What happened to "for the joy of it" as a motivation to explore stuff?

I don't disagree. I just find it amusing that scientists put so much effort into establishing something which is not susceptible to any form of verification or falsification — and therefore it isn't in any meaningful sense "scientific".

In the scientists' defence, if they have evidence that their technique has worked with a wide variety of living people, they can at least say "a technique known to be reasonably reliable has been used here, and the results are interesting, although they do not allow for every variable.

Exploring ideas is a worthwhile exercise in its own right, and often good fun — after all, that's part of why we are Forteans. It's what inspired the classic science fiction writers and many other authors and artists.

There's no reason why there should be a definite answer to every whimsical question we can come up with — and very often, even if we could, the information would be of no practical use (unlike, say the stuff we learned at school, such as the dates of mediaeval battles, or the names of Henry VIII's wives.) LOL

 

[Frideswide]

I just find it amusing that scientists put so much effort into establishing something which is not susceptible to any form of verification or falsification — and therefore it isn't in any meaningful sense "scientific".

They probably aren't. They are probably whoring what they are actually doing becuase if you don't have a certain number of "public engagement" items in each reporting period then you have problems with your institution and further afield.

 

[EnolaGaia]

The irony of "scientists" setting out to do something that is inherently not susceptible to proof or disproof. ... .

I understand your point(s), but I must submit a bit of quibbling.

There's a twilight region between "pure science" (quest for reliably demonstrable knowledge / information) and "applied science / engineering" (creation of tangible / concrete artifice leveraging such demonstrable knowledge).

Within this twilight region there may be a mixture of (e.g.) relatively "pure" scientific research contextualized with regard to practical effects (as opposed to knowledge / information creation) and / or exploratory "engineering" conducted to generate results (i.e., data) which may or may not serve to further inform more abstract "pure scientific" work.

In the case of the Live Science article (post #5) one could construe the development of a universal facial mapping template as both:

- a "pure science" exercise testing the hypothesis such a universal mapping schema applies to human features

- an "engineering" exercise to develop a means for simplifying the labor-intensive set-up work for such animations

Still, I readily concede the article's author was sloppy in shifting the characterization of the researchers from "engineers" to "scientists".