This talk was given at DevoxxUK by Jon Sneyers
Talk Summary
Images are a crucial part of any website or app. In this talk we'll give a brief history of image formats for the web, discussing both the universally supported GIF, JPEG, and PNG formats and some of the newer formats: WebP, JPEG XR, JPEG 2000, BPG and FLIF. We also briefly look at vector formats, in particular SVG and icon fonts. We will cover the strengths and weaknesses of each format and how to use them effectively.
Selecting the right format and compression quality settings for an image is not a trivial task. It has become even more challenging when progressive and responsive web design are thrown into the mix, as well as user-uploaded images. Optimizing your images is important though, since on average, about two thirds of website bandwidth goes to images. We will give you some pointers on how to do this manually, and more importantly, how to automate the process.
Jon Sneyers is a member of Cloudinary’s R&D team, and works on its image processing backend. Cloudinary is a cloud-based image management solution that manages 15 billion images; its backend processes thousands of images per second. It is used by over 100,000 web developers. Jon is an expert in image formats for the web. He created the Free Lossless Image Format (FLIF), which achieves better lossless compression than any other format.
Jon has a PhD in computer science from the University of Leuven (Belgium) and he currently lives in Brussels.
Slides available here.
The "How to"
Jon Sneyers:
Welcome, everybody, to my quick talk about image optimization and how to do it. Just a little slide about me: I am from Belgium. I work at Cloudinary, which is a image management solution end-to-end. We manage about 15 billion images. We do thousands of image encodings per second, so we are kind of experienced in dealing with images. It's what we do. I'm also one of the guys who invented a new image format called the Free Lossless Image Format.
Why does it matter?
Why does image optimization matter? 65% of an average website is images, in terms of bytes. Moreover, most of these images are part of the content of the website, just like the HTML. Unlike, say, the style sheets or the JavaScripts or the fonts, which are things that don't change very often, images do change very often, so there's not much opportunity for client-side caching of it.
A lot of the bytes that get transferred and a lot of the things that slow down your website or your mobile application for that matter are images, so if you're optimizing things, it's the first thing you should look at.
Vector images
There are different kinds of images. One type of image is vector images, which are scalable by design. These kind of images consist of drawing instructions: drawing lines, drawing curves, maybe filling them with some color.
These type of graphics can be scaled to any display resolution because they don't have the concept of pixels, so this is very suitable if you have your material in this representation and you can use an image format like SVG: scalable vector graphics, which is supported by all browsers. If you have material like this, and typically it will be things like icons, text, or plots, maybe drawings or logos, stuff like that, it could be a very good idea to use it.
However, a lot of the images we deal with are not of this kind and you cannot just convert any type of image to a vector image.
Raster images
Most of the time we are dealing with rasterized images, pixel images that are based on some rectangular grid of pixels, which is also what your screen does. Every pixel has some numeric value for the different color channels, typically red, green, and blue.
This representation can be used for any type of image, not just logos and text, but the main downside, of course, is that it's not scalable. I mean, you can always scale down an image reasonably well, but scaling it up is going to look blurry at best. I will focus, of course, on pixel images, and I will talk a bit about different kinds of image formats that are available. Here's a timeline, more or less, of the internet and of image formats for the web.
The boxes in red are pixel image formats. The ones in green are vector formats. Let's start with the ones everybody knows, which are JPEG, PNG, and GIF or GIF, I don't know. These formats are universally supported. You can use them on any browser on any platform. That's the main benefit of these formats: universal adoption.
GIF
Let's start with the oldest one, which is GIF. It dates back to the late '80s. Still, currently it's the only universally supported format for animated stuff.
If you have something animated and you want it to work everywhere, the only option still, sadly is GIF or "JIF". Either way, it's limited to a palette of only eight bits of color, so 256 colors. And it's limited in terms of transparency, in that you can only have fully transparent pixels or fully opaque pixels, but not semi-transparency. These limitations mean that you cannot really represent colors very well, so you will have to do some kind of color quantization, color dithering, which will make your images look worse.
But within these limitations, it's a lossless image format, so there's nothing lossy going on, except of course maybe you have to first reduce the number of colors.
JPEG
JPEG, on the other hand, is, by design, a lossy format, and it's specifically designed for photographic images. It doesn't support transparency because your camera doesn't produce transparency, and it's always lossy. Even a quality 100 JPEG is going to be slightly lossy.
Progressive Rendering
One of the neat features of JPEG is that it supports progressive rendering, which means that if you are loading on a slow connection, a JPEG file which has progressive rendering, it will look like the image on the right, which gives you a decent preview of the full image very quickly. The very beginning of the file already gives you a nice preview. Whereas, if you have a non-progressive JPEG, it will load from top to bottom, which means that on a slow connection, it can take a while before the bottom part becomes visible.
The progressive one is incrementally refining the quality of the image, but you can see it already looks as if it is completely loaded while it's actually still loading at the moment, but you cannot really see that. It's going to stop loading now. Yeah. The progressive one is slightly smaller than the non-progressive one. That's typical. I recommend using progressive JPEG, but there is some controversy. Not everybody agrees with me on that. I think it's usually a good idea.
PNG
Then there's the PNG file format, which was designed to replace the GIF format, mostly because there were some patent issues with GIF. It does a really good job at replacing GIF in the sense that it offers better compression than GIF. It does have support for semi-transparency and full color, but it doesn't support animation, sadly, so in that respect, it doesn't replace GIF, and that's probably the only reason why GIF is still around today.
There is an animated variant of PNG called APNG, and it is supported in Firefox, Safari, and Chrome, so it is starting to become more and more of an option. It's only recent versions of Chrome, so not yet.
How to pick the right format?
So how to pick the right image format. The first question is "Does your image have to work on all browsers?" If it doesn't have to work on all browsers, you can do much more interesting stuff, but if it has to work on all browsers, then the first question is, "Is it an animation or not?"
If it's animated, you have no other choice than to use GIF. If it's not animated, then there's no reason to use GIF. If it has transparency, then, again, you have no other choice than to use PNG because JPEG doesn't support it, but the bulk of the images on the web are still images that are not using transparency, and in that case it depends on the kind of image: which format is right. If it's a photographic image, typically JPEG will be the best choice. If it's non-photographic, PNG might be a better choice.
So it is a bit tricky to make this decision automatically, but this is more or less what you would have to do. Okay. Those are the universally supported formats. Now let's look at some of the more recent formats that do not have universal support. Let's first look at these. If you look at the support table, none of these formats have universal support. You have JPEG XR, which is supported only on Windows, JPEG 2000 only in Safari. Then there's WebP, which is supported on Chrome and Android in general also.
It's probably going to be supported soon, also, in Firefox and hopefully in Safari, so it is starting to get more adoption. Then there's some even newer formats that haven't got any browser support whatsoever.
Newer formats
JPEG 2000 is a improved version of JPEG. It adds some missing features like support for transparency. It has a lossless mode. It supports a higher bit depth for more accurate color representation. Sadly, it didn't really catch on, at least on the web.
There are some niche cases where it is quite popular like medical imaging, but on the web it's not used very much, and of course that's mostly because only Safari supports it.
JPEG XR: more or less the same story. It's an improvement of JPEG, also supports transparency, a higher bit depth, slightly better compression than JPEG, but the difference is not that huge, and it's only supported in Internet Explorer, so quite similar to JPEG 2000 in a sense.
Then there's WebP, which is a really interesting format. It was developed by Google. It's based on the WebM video codec, so VP8, which means that it has some limitations, but usually it's superior to JPEG in terms of compression quality, so it's a highly recommended format. It comes in two variants. There's lossy variant, which basically improves upon JPEG: adds support for transparency and so on. And there's a lossless variant, which is a replacement for PNG and tends to be superior to PNG. It also supports animation, so it can replace GIF.
Then there's some even newer formats. I won't go into much detail. BPG is based on H265, the video codec. It is currently the best available format in terms of lossy compression, so it will give you the best quality for a given file size. Sadly, it has some patent issues and it has no browser support, so it's not really practical to use right now, except maybe in an application not on a website I guess.
Then there's the Free Lossless Image Format.. The same situation: no native browser support, but this is a lossless format, and it's currently the best available option for lossless compression, and it doesn't have any patent issues.
Responsive Web Design
Just some brief words about responsive web design. 10 years ago it would be acceptable to have websites like this. Actually, this is an example from last week, so you can still find these kind of websites, where you've designed the website for a fixed view-port width.
You just say to people, "If you don't have this screen resolution, it's your problem." That's, today, no longer acceptable. We have a variety of devices which people use to browse the internet. People are viewing websites on very large televisions. They are viewing websites on phones and on smartwatches even. So the variety of screen resolutions is becoming huge. If you look at the top statistics of screen resolutions, you can see that the most popular resolution is actually much smaller than what used to be common 10 years ago.
It's just 640 by 360, but a bit down the list you get full HD or maybe even higher, so it's a huge range of resolutions. The question becomes how to deliver the right image to the right person because you cannot actually give the same image to everybody. It doesn't make sense to send a 4K resolution image to a smartwatch, so you have to make sure that your images get the right dimensions in terms of resolution, taking into account things like DPR.
You also have to take into account the physical size of the device. If you are showing an image on a very small device, you might want to zoom in to the interesting bits of the image, not show the entire thing down-scaled. So you need to do some kind of cropping usually to deliver slightly different variants of the same image to different people. Also, you have to take into account the browser and the capabilities in terms of image formats so you can make use of the best format that is available.
In the near future, there are going to be other ways in which different images are going to be served to different people. For instance, wide gamut displays are becoming more and more of a thing, and you might want send different versions of an image depending on this as well. Finally, bandwidth considerations are also important to take into account. In order to deliver the right image to the right person, you need to make a lot of variants of your images.
Cloudinary: "Images Solved"
Doing this manually, it's basically becoming impossible, so one way to solve this is to use dedicated image servers like Cloudinary, which I work for. Basically you just upload your original images once in the highest resolution you have available and you let the image management service do all the rest, so convert it to the right sizes, automatically crop it, automatically select the right formats, and so on and so on.
Summary
To summarize just a few take-home lessons, images are incredibly important for the performance of your website and mobile applications, but it's tricky to get it right. You probably don't want to spend too much development effort into trying to solve these problems yourself and reinvent the wheel, so I would suggest to automate this process. Use one of the existing services to take the hassle out of it and let people who know what they're doing do their thing. That's just my suggestions. Any questions? Yeah?
Q&A
Speaker 2: What are the issues related to progressive JPEG, the controversy?
Jon Sneyers:
The controversy is mostly about ... In a progressive JPEG, you first get a blurry version of the image and it gets better and better. According to some people, this can give you a bad first impression of the website because the image looks bad and you don't wait for it to fully load. Whereas, in a non-progressive you see the image load, so you know the pixels you can see, they are at the highest quality already. So it's a psychological issue I guess. There are some different opinions about what is best. There's also performance things like progressive JPEGs take a bit more memory to decode. I think most people would agree that progressive JPEG is the right thing to do. Any other questions?
Speaker 2:
What kind of optimizations do you do on the images, like stripping metadata for example?
Jon Sneyers: Yeah, stripping metadata is obvious. In lossy compression, there's many ways to compress an image. You can select quality settings, but there's various knobs you can turn. What we try to do is automatically find out which quality settings can give you the smallest size in terms of bytes but still give an image that looks good because you cannot just save everything as, say, a quality 70 JPEG. That's not going to work. Some images will have artifacts while other images, they might look fine as a quality 50 JPEG. So it depends on the image and you need some clever heuristics to figure out how far you can go basically. Speaker 3: Do you do that then?
Jon Sneyers:
Yeah.
Speaker 3:
How about with smaller images, the zooming? Actually, some kind of gravity or something like that to zoom in.
Jon Sneyers: Yeah. The question is "What about the cropping, the zooming in?" We also have algorithms for that. For instance, based on face detection and other kinds of heuristics we zoom in on the parts that are most likely to be the focal area of the image so you can automatically crop without you knowing in advance where you want to crop. We can figure out what part you want to keep and what you want to crop out.
Speaker 4:
Could you comment on what you do better than your competitors like Imgix or Kraken, technically speaking, not in terms of pricing or any of that?
Jon Sneyers:
Yeah. I don't know about pricing. Technically, I think we have the most complete solution. In terms of functionality, we have the most features. Like the automatic quality selection, automatic cropping, I think we have much more advanced features in that respect than what Imgix has to offer, for instance. I'm not saying you have to use us.
I'm just saying, don't try to do it yourself because it's messy, it's difficult, and you will waste a lot of time reinventing stuff and reimplementing things in a probably less scalable and less robust way because there's all kinds of issues, especially if you have user-uploaded images: there is also security risks involved, maliciously crafted input images and so on. We had the ImageTragick problem some time ago, so it can be dangerous to do your homegrown solutions stuff. Okay. Thank you.
