How mathematicians are helping art historians | 91TV
Transcript
- Hello. Good evening. My name is John Keating. I'm the treasurer here at the Royal Society and one of
- the vice presidents, and it's my enormous pleasure to introduce our lecture this evening by Ingrid
- Daubechies. It's the Bakerian lecture. Before I do that, I have a few prosaic matters to deal with.
- So apologies to bury myself into these. First of all, please do turn off your mobile phones,
- switch them to silent or switch them off, and refrain from taking photography or shining
- lights or what have you. That seems to be a thing these days. We're not anticipating any alarms
- sounding. So if you hear one, please do take it seriously and there are arrows that point to the
- emergency exit. So follow those. There will be an opportunity at the end of this lecture to
- ask questions, and I hope you've come with some. You can either ask questions here, there'll be
- microphones that rove around that you can use, and we have a large number of people watching
- this lecture online and for those folk, we have a Slido method of asking questions. So please do use
- that Slido technique to send your questions in and they'll be asked in the room. Those things,
- having got them out of the way, now it's my great pleasure, as I say, to welcome Ingrid Daubechies
- to give the Bakerian Lecture. The Bakerian medal and lecture is the premier lecture in the physical
- sciences within the Society. It was established in 1775 by Henry Baker, who, with a gift of £100, and
- treasurers before me have clearly done their job well because the award now has grown to £10,000,
- and there's a silver gilt medal that is awarded. So this really is an award that we
- esteem very highly within the Society. Tonight's lecture will be entitled How Mathematicians are
- Helping Art Historians and Art Conservators, and will be given by Ingrid Daubechies FRS, as
- I indicated. Professor Daubechies holds the title of James B. Duke Professor Emerita of Mathematics
- and Electrical and Computer Engineering at Duke University. She started her career in
- mathematical physics but then branched out to signal analysis a few years after her PhD. She's
- famous for her construction of bases of wavelets, which are supported on finite intervals. These are
- techniques to analyse data. These not only solved fundamental mathematical problems but have many
- applications across science and technology. For example, they play an important role in remote
- sensing, medical imaging, and digital photography. She's introduced many sophisticated mathematical
- techniques to analyse these approaches to signal processing. So it's an enormous pleasure for me to
- welcome Professor Ingrid Daubechies and to invite her to give this year's Bakerian lecture.
- Well, it's an enormous pleasure for me to be here, and I've already eaten six minutes into
- my time. So without much further ado. So this is an overview of what I want to tell you about.
- I'll start with a very early application, late '90s, early '00s, not by myself,
- then give you a very impressionistic primer to what wavelets are, and then talk about some
- applications I have been involved with myself. For those of you who've learned the abstract by heart,
- I apologise for skipping some of the topics that were described there. You can find
- other talks by myself on YouTube, but I will not be talking about role mates in Van Gogh paintings
- and so on, because I want to get to the last two applications, which were in collaboration with
- people at the National Gallery here in London. This is a list of many people I have worked with
- on these topics and others. I like to give it at the beginning, because at the end, I always
- have too little time and I don't want to skip them. So please. Okay, restoration of frescoes
- by Mantegna. The Eremitani Church in Padua looks like this, and at the end of the Second World War,
- this is what it looked like. It had been bombed because it was very close to a military target,
- and bombing was not a very precise tool, and some fantastic frescoes by Mantegna were reduced to,
- of which you see a reconstruction here and there, were reduced to this kind of rubble. It
- was near the end of the of the war in the Italian theatre and they were all gathered up very quickly
- by the big Institute of Art Conservation in Rome, and they had these crates and crates of fragments,
- and every so often they would open them and sigh and close them again. But in the late '90s, they
- thought that maybe it might be possible using by then modern computation tools to help put
- these frescoes together again. So they cleaned, stabilised, catalogued everything, and they put
- them out on CD-ROMs. The older ones among you may remember CD-ROMs. So they sent out… It was a big,
- big challenge because there were a large number of fragments. They covered a big area together,
- but it still was less than ten per cent of the total area of the puzzle. So imagine that you
- have to put together a puzzle of which you only have ten per cent of the pieces and not only is
- their location unknown, but you don't even know in what orientation to put them. They
- did have the equivalent of the box that you have when you have a jigsaw puzzle, because they had
- high resolution black and white pictures of the originals, and they had very precise measurements
- so they could correct for the parallax. So they actually did make printouts at real scale of the
- originals, but then you still had the problem of where are you going to put these pieces. So
- they needed a fast method for 80,000 fragments to put them in place and they asked for
- different proposals. The proposal that won was a proposal by Massimo Fornasier that I will describe
- in a few slides. Okay, so the digitised images, and this is something that we will encounter over
- and over again, digitised black and white images consist of a lot of numbers. For every pixel,
- a pixel corresponds to a little square that has a constant grey level, and those are
- numbered from zero for all black to 255 for all white. That gives you 256 different levels, and
- that is what then gives you the grey index in between. Now, if you rotate, the thing is that the
- numbers look completely different. So you can't just say, look, I have these numbers, let's just
- not rotate. The thing is very different. This is just a sketch of how different, but in fact even
- worse. If you move it slightly as you try to make the grey levels, the numbers look different again.
- So you can't just say, let's take this table of numbers and where do we find those same numbers?
- I mean, it's hopeless. So Massimo came up with a method, that was the method that was used to
- recognise rotations. He said, let's use circular harmonics. So circular harmonics are very special
- functions that live on the disk. I mean, it's shown here as squares because digital images use
- these rectangular pixels, these square pixels, but you should really think of the circles. So these
- are examples of these basic image functions. I mean, rendered as images with large and small
- values, but of these circular harmonics, and to show you that indeed, I can decompose anything in
- circular harmonics, let's look at the image at the bottom right here inspired by one of these
- frescoes. What's shown at the top as we go left to right and then down and so on, as you would in a
- text, is the additional building blocks, these circular harmonics that we saw in the earlier
- slide being added with just the right magnitude in order to together build up… Sorry, that was
- not what I wanted to do. I don't know, somebody somewhere has to do something about this. Maybe
- I can… Oh, there, thank you. So as you add them, you gradually build up that image at the bottom
- right. So all these building blocks with exactly the right amount build up to this. So once you've
- decomposed an image in these circular harmonics, it turns out it's very easy to then look at what
- would the representation be of a rotated one. So you have the original image in each one of these
- harmonic functions with the right weight, a little bit of this, a third of that, maybe only a quarter
- of that, and so on, adding them. Then the rotated version uses exactly those same weights with an
- additional rotation factor, because the original functions here have a lot of rotational symmetry,
- and that expresses itself by saying that the rotated function gets a rotated vector, which is
- different for each of the harmonics, but we know if you have this angle, then I can compute all
- those rotated numbers. So then you can see how the numbers would change for the full reconstruction
- and you can do that very, very quickly. This was then used to put together a facility in which
- volunteers went through all those fragments, and you see how old this picture is. I mean,
- this is what terminals looked like at that time. I mean, it's been a while. So they came and every
- volunteer would get, from the database, would get an example, would pull it up, would adjust to
- the biggest possible circle that could fit within the fragment, that determined the radius at which
- they had to search for circular harmonics in the whole thing. These had been precomputed for many,
- many different radii. So they could then look at the numbers and the rotated versions of those
- numbers, which they could do with the circular harmonics, and they could then produce what
- were the most likely answers. Typically in the top three, it already contained the right one.
- When not, they could pull up the top ten, and in the vast majority of cases, the top ten
- contained the right one. The final decision was done by a person, but that was fine. Actually,
- the art historians and art conservators preferred that it was not something that spewed out answers,
- that there was some human judgement in that, and it makes the algorithm much easier. Usually,
- you don't have to work to get it 100 per cent. I mean, the high percentage within a final human
- decision is usually preferable, easier to code and preferable for the user. Shoot, what did I do now?
- Sorry. Magic please. Okay, so here, you see an example of fragments that were placed on the right
- picture, and here is an example, in the little red square here, you see every time where the location
- is of the detail and the fragments placed, and we'll see a few of those, and you see how really
- precisely they could put all these fragments. You might say, but what's the use of that? You only
- have ten per cent of the full area. Well, this type of information gives art conservators a lot
- of - because they knew the grey value thing, but now they know what the colours are for the toes
- of this soldier and his leather leg protectors, and the colour of the lance haft and so on. So
- this is information with which they can complete colours in paint, it propagates colours. Here
- is another one with fragments placed. This is a way in which this particular panel,
- this particular fresco was made in the restored chapel. They chose in some places to
- show the original colours, but with a slight tone emphasis showing the difference between original
- fragments and reconstructed, and in others they show the grey scale to show what was
- the information that had been used to make that reconstruction. Here's an example of a picture
- of the chapel as it was being restored, and you can now go to Padua and view this restoration,
- but in fact, what techniques do the restorers use? They propagate this. They have information
- in some places, and then they propagate it by eye, by there, and so on, but propagating,
- we can do that mathematically. We have equations that flow stuff. We have beautiful ways of doing
- it. So we could do that mathematically. I mean, this information you can flow and get the full
- reconstruction, and in fact, that's something that Massimo is now hoping that he's building,
- he hopes to build something that will people have as an app on their phone where they could see the
- partially restored frescoes but see the full thing as it would have been. So this is the first of my
- chapters. Now I want to go… The important thing, I mean, there are many, there was a flowing of
- information to propagate the colour, but the important thing that Massimo proposed was the
- fact that he used building blocks that made it easy to talk about rotation, because that was
- the problem that challenged there. So now I want to tell you about different ways of decomposing
- images that answer different questions and give you this very impressionistic primer on wavelets.
- So as I said, pictures consist of small squares - digital pictures consist of small squares
- which have which have numbered in grey value from 0 to 255. Let's take this example of a
- greyscale picture of a portrait, self-portrait of Van Gogh. If I enlarge this and enlarge it again,
- you start seeing the individual pixels, and these are the corresponding greyscale values. I have put
- the ones below 100 in bold to remind you that the darker, the smaller the number. Let's take only a
- little bit of them, but you should really imagine the computations I'm going to show you happening
- all over the image. What is striking about these numbers is that very many of them are very similar
- to their immediate neighbours. Not all, but in fact, this happens to every digital image that
- you might take with a photograph. Imagine taking a photograph of the front here. You would have that,
- as the grey in your photograph changes gradually here, the numbers slowly vary, but they're very
- similar. Of course, here, there's a very sudden transition between a dark and a light. This
- transition could happen anywhere. Nothing says transitions can only happen at three quarters of
- my image, but they happen rarely in every single image, and that's what wavelets exploit. So
- what we're going to do is we're going to take these numbers in pairs and for each pair compute
- their average. We only have half as many as we had originally. Again, they're very similar. We can
- compute their average. That gives us only four for the original 16. We have lost information.
- If you know the average of two numbers, you don't know the difference between the two
- levels anymore. So let's compute every time also what that difference was. If I know all those red
- differences and the four averages that I obtained after two levels of averaging, I can go back,
- because if you know two numbers, you know their difference and their average, you can go back.
- But many of these differences are very small. It's only where something was happening in the image,
- this sudden transition between light and dark that a large difference shows up. So I can forget
- when they're small and the algorithm has told me where I will need them. We'll do this, of course,
- in images, you have things in two dimensions, not only horizontal. So we have here, numbers
- in two directions. If I take those numbers then I can take them in pairs on… Oops. Oh, please.
- I can take them in pairs and take an average and a difference, and the next pair, an average and
- a difference, and I can keep doing that. So I have now two arrays of numbers for each row,
- I have only half as many numbers in the average, and half as many in differences as originally,
- but I still have 16 numbers in total. On each of these, I have nothing done vertically yet,
- so imagine doing on the same thing vertically, averages and differences, and then on the
- other array I have to do the also averages and differences. So I have now four different species
- of things, averages in both directions or things that have at least one direction of differences.
- What do they stand for? Before we saw that, there was a sharp transition between dark and light,
- gave us a big number. If you have big numbers in these here, imagine that you had something
- with horizontal stripes. So very small numbers, very large numbers, and so on, and I've taken
- averages in differences. Different averages, very small, very large. Then I start taking
- differences vertically. All of a sudden, I'll get big numbers. So horizontal stripes will give
- me big numbers in this thing, where I've averaged horizontally but different vertically. Similarly,
- if you reason it through, vertical stripes will show up in big numbers in this array, and then
- this way, you take differences in both directions. Well, that's why you have a checkerboard kind of
- thing. You have big differences and then big differences in the other direction. That makes
- an even bigger difference when I do it. So let's see what this does on a real image.
- This incidentally shows, it doesn't have to be a square image to begin with. Averaging both for
- each pair horizontally and each pair vertically, I compute one average. Four becomes one. So I get
- something that's half the size in both directions and it looks like the original image. I'm going
- to now use the correspondence between numbers and greyscale values to show it as a picture.
- When I take differences, however, numbers from 0 to 255, some of them may be negative,
- so I don't know how to do it. So what I do is I say, well, it can go from negative to 55 to 255.
- So the middle grey now will be zero and the dark things will be negative and the light things will
- be positive, and as it turns out, our eye is much more sensitive in a field of grey to the dark than
- to the light. So you will see really only the dark things show up, but you see middle grey almost
- everywhere and a few dark things for vertical lines, as I predicted. Here, we do difference
- but averaging difference in horizontally, we get horizontal features and oblique features, but a
- lot of middle grey. Let's do it again, because the top left here is just an image again, I can do the
- same thing. On this image, it's average and the three species of difference, and they only add up
- to what that first thing, that was half a quarter of the full image had as information. So I have
- to add these layers to get the full information. And again, and again, and I believe again. Okay,
- so I've done operations here that give you the full information of the original image but in a
- transformed way and in a transformed way, in such a way that many of the numbers will now
- be close to zero. Let's understand a little bit more about this. Let's visualise. I mean, I can
- go back. I promise you, I always can go back. So let's go back but not add all the detail. So if I
- go back from that top left thing and I forget all the details from these, these other detail things,
- I get the thing on the left, and this shows you that I have swept something under the rug here,
- because if I was really, truly taking averages and differences and I forget about all the
- differences, then on the left I would have now big blocks of 32 by 32 pixels that would be uniform,
- the same grey level. I don't. That's because I did something smarter. I used something
- that's morally like a difference, but of higher order and so on, and there's some mathematics,
- lots of mathematics that comes in, but I'm not going to go into technical detail, but morally
- it's still the same thing, but it will give you something much smoother. If I add more detail,
- you see on the left, reconstruction becomes and so on, and more, and more, and more, and so I have
- the full image. Let's look at two separate details to make clear what's happening. Let's look at this
- thing in the sky. So we concentrate on the patch of scale, and what I'll do is I'll show you the
- successive reconstructions. First, very bad, and more, and more, and you see details coming in,
- but I need all those layers in order to get there. But let's now look at the sky. I'm sorry about the
- bottom, look only at the sky. There's not much that's going on there. I mean, it's perfect,
- very soon. You don't need all the extra layers. That's because there was no detail there. So if
- I forget all those coefficients, the ones that were near middle grey, and so you can use that
- for compression and that's what's done. So the left is the original image. The right is that
- same information all the information but in a different form, but I can choose to take only
- some of the information, only the pixels that I've painted red here, and that will give you
- the image on the left. Now it's not perfect. Look, I'm toggling back and forth. It's not perfect, but
- it's pretty darn good for something that's only three per cent of the full information. If I take
- actually a little bit more, then you get a perfect image. So this is how you can use compression, but
- because I was so very localised, what I can also do is I can say, look, suppose I have a thumbnail,
- and I know there's one area of which I want the full detail. You can have, imagine applications
- where you have medical data over a very, very small pipeline, and you want detail in one region,
- but you don't want to spend ten minutes for every picture to upload it in full detail. There's only
- one area that you want. So you paint it red and you say what the coordinates are. That enables
- the database to compute that you only need those pixels and you get only those, and you build that
- area with full. So that's the power of this mathematical tool. What I have done is I have
- really decomposed the image because I was looking at small scales and then bigger and bigger and
- bigger by making my algorithm, by taking averaging over larger and larger. I have decomposed into
- building blocks that were very tiny or much larger. The standard way of compressing images
- uses this kind of building blocks. They use little squares in your image and then use something like
- circular harmonics, but now for the square. So they all have the same size. I'm not going
- to talk about any more mathematics than this, but I want to tell you a little bit, because
- now I'm going back to applications. So I've been working on applications that are related to high
- resolution images of art pictures for the last 15 years or so. I worked with the Van Gogh Museum on
- some early questions, which were actually not the most interesting questions that I've worked on,
- and then that led - every time I did something and I would give a talk about it, people would
- come up with new questions, and so at some point, I had a sabbatical in Belgium and I met
- Maximiliaan Martens, who is an art historian at the University of Ghent and he asked me, he said,
- oh, wow, if you could do this, maybe you can help us with this question, this challenging question.
- It was a question concerning the altarpiece in Ghent, which is an extraordinarily famous picture.
- I mean, of course, having grown up in Belgium, I knew of this picture. I had taken school trips
- to this picture, but it's also really, truly an extraordinary picture of which you can find
- extremely high resolutions online. I encourage you to look for it because you can look at zoomed
- in images of a quality that are incredible. Jan van Eyck was an incredible perfectionist. I mean,
- if you zoom in on that picture of the Lord in the middle there, then you actually find that he
- painted individual hairs of his beard, he really did, but the controversy that Max Martens told me
- about was about, this is a polyptych, if you close the doors, he said they had a discussion about
- that book here in the Annunciation scene. He said, well, van Eyck was such a perfectionist that some
- art historians believed he really had painted a book, a text of a book, and others said, come on.
- They said, well, you know, the hairs of the Lord, I mean. So he showed me the picture. I said, well,
- these books, these kind of medieval letters, which I learned are called lutena formata,
- I can't read them, I can't read them, not even a reel in a museum, I said, but you can. So why is
- it a controversy? I mean, you try to read it and if you can read it, it's a text, and if
- you can't read it, it's not. He said, well, it's not that simple because there's all these cracks,
- that makes it very hard for us, because it turns out our visual system is extraordinarily good at
- detecting patterns. I mean, we see channels on Mars. We see the man in the moon. We see patterns
- because we are good at finding patterns in images. We're extraordinarily bad at thinking things away.
- I mean, once the cracks are there, thinking them away is hard. So he said, because we can't think
- those away, if we could think them away, maybe we would be. He said, well, math can help me. So
- indeed. So he was showing me letters where people doubted and so on, and I said, well, get me high
- resolution images. It turned out it's much harder than I thought because of course, the image itself
- has lots of the same browns as you find in those cracks. So to decide whether something is a crack
- pixel or not. I mean, the idea was you just find what pixels are cracks and then you mark them,
- and then you use the inpainting techniques to inpaint them, and then you try to read it,
- but deciding what pixels for cracks was tough. It became about a third of a PhD thesis in Brussels.
- The key was finally not to decide, not to find the optimal method, but to find several methods
- that were pretty good, and then you could take a majority vote of those, and that gave a very
- good method. So Bruno Cornelis in Brussels went from the left to the right and I thought when he
- showed me, he was so proud, and I looked at it and I said, well, fat lot of good that has done,
- but then the experts said, oh my, look at that. I said, yes? And they said, oh look, this is a W,
- and so on, and they were so excited because they felt they could recognise several groups
- of letters that led them to conclude, not only was this a book, but they became convinced what
- book it was, and this was a book that had been copied in Bruges ten years before and so on. So
- they were really excited. So big relief, but yes, mathematics could help with that. That actually,
- a talk that I gave about this led to the next application. So this is about a much,
- much, much less famous altarpiece, of which three panels were in the North Carolina Museum of Art.
- One was in the Portland Art Museum. Three are in the Metropolitan Museum of Art, and the central
- one was in the Art Institute in Chicago. Now they had been separated late in the 19th, early 20th
- century. They had been identified only in the '80s as belonging to the same altarpiece, because once
- a church got decommissioned, which happened to this church, and the altarpiece gets sold,
- art dealers could get, of course, more for one altarpiece than for small panels,
- but more for nine small panels than for one altarpiece. So many of those were then
- sewn into pieces, and they had found their different ways to different museums, all in
- the US, by coincidence. So no border crossings were necessary, so that was already great,
- but they had then undergone different conservation history and so on, and here are the photographs,
- were the photographs by different photographers in the different museums. When David Steel,
- the art curator at North Carolina Museum of Art found by detailed technical analysis that they
- all belonged to the same altarpiece, he had dreamt of bringing them together, and the museums were
- saying, yes, okay, well, but, you know, it's not complete, you're missing one panel, and they said,
- what's the point then? So David Steel commissioned from Charlotte Caspers, who is a fantastic artist,
- who is an expert at… Although her goal was to make a career as an artist, she wanted to learn
- much more about materials and how they work and their intrinsic properties than she felt
- was typically taught at art academies, and so she underwent training as an art conservator to get
- all that training for materials. As a student, she then made a beginning of a career as an art
- reconstructionist. That's to say she would create copies of art, but using the same techniques and
- materials and everything as the original artist would have done, and museums loved her work
- because it was very high quality, but because then they had something that they could let high
- school students handle. I mean, that she's copying here in Maastricht is, of course, priceless and
- people cannot be allowed close to it. It also was very useful for scientific study. So of course,
- in the case of NCMA, there's a missing panel. It's not a panel next to it that she can copy,
- but they had a very good idea of what the missing scene was because the panels here, the little
- panels tell the life of John the Evangelist and not just his life, but his life, as described in
- the Legenda Aurea, the Golden Legend, which was a book that was enormously popular in Middle Ages.
- We have zillions of copies of it, so they knew what the missing scene was. It was the baptism
- of a heathen priest who had challenged John, if your God is so powerful, why can't he do this
- and this and The Lord had accommodatingly done all the miracles, and then the guy wanted to be wanted
- to be baptised. So that baptism was the missing scene. So using the pictorial vocabulary of the
- other panels, Charlotte and David Steel made a composition of that mixing, because they were
- sure if the panel is ever found that it will be different from that panel, but they wanted to make
- something that would not have surprised in the 14th century. So here, you see her, because she's
- borrowing elements from the other ones. Every time she would take the element that she was borrowing
- here, the robe of John from the panel from which she was borrowing it. So she made this fantastic
- panel. It's beautiful. It's an object of beauty. It's bright, it's shiny, because these things had
- polished gold backgrounds that with punch marks, gave vivid reflections. They made a documentary
- of its realisation to show at the exhibit, and then they realised they had a problem because
- they couldn't put it next to the others, because then everybody would only look at that one,
- it was the only non-authentic one, and plus there would be such a contrast. Well, math can help. So
- because we had the high resolution pictures of the others and also the new one, we could do a virtual
- ageing of this very new one. It wasn't going to be physically touched, and a high resolution printout
- could be shown next to the other ones, and then the new one could be, and that's what was done.
- So actually in this photograph, the ninth panel, the printout of the ninth panel stands out more
- than it did in the exhibition hall because in fact, we only noticed it afterwards, but the
- reflection properties of the paper underneath were sufficiently different that they show. In reality,
- they didn't show at all. In fact, when the curator came from Chicago accompanying the central panel
- to install it for the exhibit, he was angry. He said, you guys made a forgery here. I said, no,
- it's printed on paper, look closely. So it had fooled from a little distance even an expert.
- But you can then, of course, do the reverse on the old ones. You can rejuvenate them, and so that was
- done. What do you need to rejuvenate? You need to get remove the cracks, but we knew how to do that.
- We learned that in hand. You have to do colour correspondence. We learned them in one direction
- so we could do it in the other direction, and then we had to rejuvenate the gold. So cracks, it's not
- easy, but we knew how to do it, and indeed the crack map method that we had developed beforehand
- worked for this altarpiece by Francesco Gissi, and we could inpaint, and here actually, you see,
- we did it very gradually. At every intermediate stage, we actually asked the art conservators
- and the art curator, can we go one step further, or should we stop here, because we didn't want
- to make it a chromo, we wanted it to be… So in collaboration, we stopped colour remapping. You
- see, there are things that are very similar. Here, we had the new colour for John's robe
- here and the old colour for John's robe. So for many of the colours, we had such correspondences,
- and so let me show, I'll take away the gold first, but if I take the old colours of the eighth panel
- and rejuvenate them, I get, and if I take the new fresh colours of the ninth and I age them,
- we could do that. Then in order to make the ninth panel old, we had to put in cracks, but
- we learned cracks are very different depending on the pigments, but we had learned that in our study
- and we had to add gold, which we stole from other panels and just the images, of course. We still,
- in order to rejuvenate the other, this was what was then shown in the exhibition. To rejuvenate
- the gold turned out to be a new challenge, because how do we see that something is metallic? Well,
- if it's something that's like a spoon, you know what spoons look like, and so you see, this spoon
- looks silvery and that looks only stainless steel like, and so on. If it's a flat thing,
- it's more challenging, but you still move your head a bit, you move your eyes a bit and so on,
- and that gives information of how things look from different angles and your brain integrates that,
- but we don't have that when you show just a picture. I mean, reflections normally don't change
- when you move your head. So what we had to do is we had to render it like Pixar makes its movies,
- rendering surfaces. So you made a model of it that was a flat object with the painted surfaces,
- and the rest was a metallic thing, and then you showed it, showed it moving or seen from different
- angles, and for that we used computer graphics methods. So there was not only the polished thing,
- but there were also these punch marks, which are very typical for this period of Italian painting.
- Charlotte, to make her copy, had found the punch marks, the shape of the punch marks,
- had them constructed by an art history student who worked on metal and had used them. Once we
- had the shapes, we could find them, and then we had a map of where those punch marks had to be,
- and we could use that to make a little movie. So I'd like to switch here to the movie. So the
- movie that we're going to see is one that was… I didn't do anything. The movie that we're going to
- see is the one that was shown in the exhibit. So the exhibition, you had the altarpiece with the
- new ninth panel, you had the true, but virtually aged, you had the true ninth panel with materials.
- Then you had a big screen with the documentary, and then on the final wall, the fourth wall, you
- had a big screen that showed this rejuvenation, and you see the reflection. Now, there may be
- some among you who say, well, we can make better reflections than that. All the work was done,
- all the analysis work and image synthesis work done in this exhibition was done by undergrads,
- because all of it was stuff that was no longer a research project, so I couldn't have my graduate
- students work on them or postdocs. But undergrads learned a lot from it, and they were really so
- excited to work on these beautiful pictures rather than on the standard images and image analysis,
- or a red and a yellow pepper, for instance, or worse, and this was absolutely wonderful.
- We had little videos on tablets that showed interviews with the students saying what they
- had done and how they'd done it and how much they enjoyed it. And this exhibition became enormously
- popular. They thought it would only attract a very small public of experts, but the docents loved it,
- the young people loved it, and so on. So this was a wonderful thing. Can we go back to the slides,
- please? So we have had our movie. I want to talk about, very briefly because we started a
- few minutes late, about some subjects, some topics that that came later. So after this
- project with NCMA, I had the pleasure of working with researchers here in London, many of whom are
- present, in information theory, in image analysis at the National Gallery. So these are two of the
- topics that we worked on with them. X-ray image separation, we worked on it for paintings at the
- National Gallery. The one I'm going to show you is again the Ghent Altarpiece, but you see the
- Ghent Altarpiece, I told you, it's a polyptych. So these doors have things that are big wooden
- panels that are painted on both sides. This is the other side. These are X-rays of those panels. Now,
- you might think, what is this nonsense? Well, and this happened to many panels, not only things of
- polyptychs. It's called cradling. In this case, it happened because in the 19th century, there
- was a time when the polyptych, all the panels were exposed in a museum, and they wanted to show both
- sides of these doors and what they did, the mind boggles, but they sold it through. So they had
- done both sides, but once you do that, you lose the structural integrity of these planks. I mean,
- they're really planks about an inch thick. So in order to then give more structural integrity, they
- would put a lattice of hardwood on the back. This was before people knew X-rays was an interesting
- tool, a useful tool for art conservation. So once you X-ray these things, all you see is these
- letters, and actually a completely different project we have is we developed a software
- that will remove these artefacts from X-rays. I mean, that's a different talk, no time for it,
- sorry. Adam and Eve were considered a bit too scantily clad to be exposed in the 19th century
- this way. So they had copies painted with some more strategically placed pieces of cloth, and so
- they didn't have to sew them in. So those have not been cradled, but the result is that when you take
- an X-ray, you see both the front and the back. So Adam and Eve, and these are the panels on the
- other side. Now, if you think of taking an X-ray, then one of them you'll see straight and the other
- you'll see in reverse. So let's reverse them. So these are the two, and on the X-ray indeed,
- you see Adam's eyes coming through, and you see the folds, and here you see Eve's shoulder and the
- folds again, and her eyes nicely shining through the folds there. Art conservators have a lot of
- experience reading these X-rays for very detailed information. It's hard to think things away. They
- said, can you, given something like that, can you invent virtually, virtual X-rays, that would be
- the X-ray if it only had one side? That is a hard question, and we worked hard on it. So here again,
- a detail. You see Adam's eye and the ropes. Again, you see Adam and the ropes. So we used various
- approaches and, in the end, the approaches that work best use neural networks, which are a fairly
- recent development. So I'm going to just show you examples here. We have Eve and the folds that are
- superposed in the X-ray there, and this is the decomposition. It's not perfect, you see. I mean,
- we have a little bit of fold here, but actually people in Brussels in the Art History Institute
- that is interested in these things were very, very pleased with the results, and then they
- wanted to work on the full panels for this. Finally, in my last one and a half minute,
- hyperspectral imaging. So this is a painting in the National Gallery which has two… I mean,
- it was customary in that time that a master would ask his students to work on some of the painting,
- but this was an unusual painting in that this master had an extraordinarily talented student who
- he probably entrusted with much more than details. So you have the hands of both Lippi and Baldelli
- in this painting, and there was a great interest in finding the underdrawings. So when a painting
- like this is made, typically on the blank canvas, the artist will make underdrawings that guide the
- people who then paint, himself or herself, or a student, and were there different styles in
- the underdrawing? How did it change? And so on. So the underdrawing was very important,
- and the underdrawing is something that you can see, I mean, because it shines through, but you
- can see it in different wavelengths more clearly. So here, you see for instance underdrawing,
- and you see things that have been slightly changed in the composition, in the underdrawing,
- you see there's a hand here on the back which is not reflected in the final painting. So it
- turned out that different wavelengths showed different underdrawings better,
- different pieces of the underdrawing. So there was this whole looking through it,
- where do we see this, and so on. So you would have to look through this whole different, this cube
- of hyperspectral imaging, and experiments were done and this illustrates that some
- underdrawings show up in some and not in others. Here, again, this shows up better here than there.
- Do I find another one in the reverse direction? Yes, this one is clearly much clearer here than it
- is there. So the end result, sitting here, to name him, who is now a graduate student here in the UK,
- actually developed a method to look, with collaborators, to look through this hypercube
- and find out all the different details and integrate them in one image that would then
- be easy to read. I'm going to stop here. This is the last one of my slides. I hope I've given you
- an impression of interesting, very challenging questions that you can find from art history and
- art conservation and that mathematicians are helping solve them. Thank you.
- What a lovely lecture and what interesting, fascinating material. I think we have time now
- for questions. If you're in the room, please raise your hand and someone will bring a microphone to
- you. If you're watching this online, there is that Slido technique which apparently,
- we can get to work. So questions, one there and then the next one over there.
- Thank you. How do you know which mathematical technique works best in any situation in
- advance? So for the Ghent Altarpiece, you said deep neural networks work really well.
- Do you know that beforehand, or is that just a question of trial and error?
- No, you don't know beforehand. What we do know, so I believe in all the
- interdisciplinary projects on which I've worked, whether it's in art or other fields,
- that it's very important to be in a constant dialogue with the people, of the expert or the
- experts of the domain. So you don't just get a question, go away and then come back six months
- later with an answer, because it's typically the wrong answer because it was the wrong question.
- So there's a whole dialogue about identifying and trying to put your finger exactly on the question
- and articulating answers and say, if I can do this, will that help you? And they say, no,
- I forgot to tell you this, and they tell you then something that they all learned in kindergarten,
- but you went to a different kindergarten, so you didn't know. So you learn more about the nature
- of the question and that then suggests some tools to use, and then you try, and then you come back
- with partial answers. I like to do it in such a way that I have answers that already exist, that
- are not the final answer, but that I can put back in the language of the expert to tell them, look,
- this is an intermediate result. Do you think we are on the right track? Is that something that's
- already partially informative to you? So that's why it's called research. We don't know. I mean,
- when we know, we can put undergrads on it. Yes, please.
- That was a fascinating lecture, but I do really have a problem with the altarpieces,
- the missing altarpieces. I mean, considering that it's a piece of living history, as it were,
- that it's passed through many hands, and the artist who painted it had a particular meaning,
- he had a particular understanding of what the scene meant to him and why. Obviously,
- he was communicating with other people who knew about the Bible at that time. I mean,
- would your techniques work on say, other cultures that have a different tradition of art?
- So in the reconstruction, in the whole arc of what was going on there, there was first
- identifying something that would really have fit in the artist's intent and cultural and pictorial
- tradition and trying to make something that would fit in that, and that is what Charlotte Caspers
- and David Steel did. I mean, the curator and the artist, to make something, and then we used
- what we knew about this art, namely we actually used for the colour reconstruction that at that
- time of painting, there was a technique in which for every, for instance, for the rope of John,
- the artist would have made a colour and then added a lighter colour to make the highlights to that
- one colour he'd chosen, or added a darker colour for the folds. We used that in our mathematical
- remapping of colours as well. So we used all the knowledge we had. If it were for
- a different culture, different art, I know of some Chinese engineers and computer scientists
- who have worked on Chinese art, on paintings on silk scrolls, on rice paper that's then applied
- to silk scrolls. I think it's good to have people who understand a lot about the techniques. That
- doesn't have to mean that you're necessarily of that culture, but yes, the technique and
- so on plays a role in building the tool. Excellent. Any other questions? There's one
- down here and then one in the middle there. So whoever gets the microphone first. There we go,
- you go first, and then there's one down at the front here.
- Thank you. I was just wondering when you approach these problems, do you find that you need to do a
- really bespoke solution that really defers to the specific physical history of the object
- you're looking at or can you develop a solution and then find that it can be generalised out to
- other applications? I know you mentioned with the cradling that that's one that you have refined,
- but do you find you need to be more bespoke or more generalised?
- So in the very end, it always becomes bespoke because the very final details have to be bespoke.
- On the other hand, I'm always interested in problems that are not just solving one particular
- issue. So in the cradling, for instance, the first time I saw the cradling was on the Gissi
- altarpiece when we didn't use the X-rays, so I didn't show you, but I did ask to see all the
- information. When they showed me the X-rays, those had been cradled, not because - they were sawn in
- two, not because there was something else on the back, but because the collector who had collected
- it didn't want these thick plank pieces working under temperature changes on his walls. So then
- they sawed it to a couple of millimetres and put hard wood lattice on the back so that it wouldn't
- work as much. I mean, this is a technique that's no longer used in conservation because it puts
- a lot of strain on the wood that has the art. So when they said could you remove that, and I said,
- well, we could, but I thought, well, that's a lot of work because I could see it was not going to
- be easy, but maybe it's just for this particular case and we don't even need the information. I was
- at the Prado and I talked with Laura Alba, who is the director of their imaging centre, and I said,
- do you have this too? And she was, I mean, very, very Mediterranean,
- she said, do I have this too? And she pulled open a drawer. She said, look, look, look, look, look,
- cradling everywhere. So that's when we decided to develop an algorithm for it. So I tried to,
- but in the very end, the very details depend on the practical case. The Ghent Altarpiece
- has cradling that is nicely rounded, for instance. So you need to model that rounding. It's not the
- same in all of them. So some bespokeness, but I like to look at things that have more general.
- And then a question just here, I think. That will have to be our last question.
- Hi professor, I'm so excited. My name is Alicia. So I'm the Chinese,
- you mentioned working on the Chinese art. So my question. So currently, I'm using
- algorithms training over almost one million data sets to train calligraphy, Chinese calligraphy.
- That would be wonderful. One problem we have found with applying methods of training on art, I mean,
- of using AI methods that work on, like ImageNet and so on, and using those methods for art is that
- our data sets are much smaller. I mean, ImageNet had the whole, or Google uses every
- image on the internet to train their models, and we typically have much smaller. I mean,
- we have very high resolution, so we can use patches and that's typically what we do,
- to get more examples. Chinese art, well, Asian art in general has also this fantastic property that
- there are many copies of masterworks that have been preserved, and so that's something also in
- which learning could really be exploited, but I haven't worked on those myself. I've encouraged
- some of my students, my Chinese students to get into touch with museums, because the other thing
- is you need data. You need very high resolution data. I mean, we need, for everything we've done,
- we need very high resolution data, much higher resolution than when you go to a museum and
- you take a picture with your smartphone. Well, thank you very much indeed. As I say,
- I'm very sorry, but there'll be a chance for informal questions later,
- but it's now my pleasure to formally announce that the Royal Society Bakerian medal and lecture is
- awarded to Professor Ingrid Daubechies for her outstanding work on wavelets and image
- compression, and her exceptional contributions to a wide spectrum of physical, technological,
- and mathematical applications. So Ingrid, my warmest congratulations and my thanks for this
- utterly splendid lecture, and now I'm going to present you with your award and your medal.
- Thank you. So if we go to the centre
- here. So many congratulations. Thank you so much. Thank you.
- Can I show them? I want to show you. Oh, this is complicated.
- Hopefully not something that needs to be reconstructed mathematically.
- Good. I'm now going to invite you to join us. We have a drinks reception out through the doors at
- the back there. I look forward to talking with you all and I'm sure Ingrid does as well. Thank you.
Join us for the Royal Society Bakerian Prize Lecture delivered by Professor Ingrid Daubechies ForMemRS.
The Bakerian Medal and Lecture 2025 is awarded to Professor Ingrid Daubechies for her outstanding work on wavelets and image compression and her exceptional contributions to a wide spectrum of physical, technological, and mathematical applications.
Professor Daubechies will discuss the recent years where mathematical algorithms have helped art historians and art conservators put together the thousands of fragments into which an unfortunate WWII bombing destroyed world famous frescos by Mantegna, decide that certain paintings by masters were “roll mates” (their canvases were cut from the same bolt), virtually remove artefacts in preparation for a restoration campaign and get more insight into paintings hidden underneath a visible one.
About the Royal Society
91TV is a Fellowship of many of the world's most eminent scientists and is the oldest scientific academy in continuous existence.
/
Subscribe to our YouTube channel for exciting science videos and live events.
Find us on:
Bluesky:
Facebook:
Instagram:
LinkedIn:
TikTok: