Here, in Part 1 (of a three-part series), we discuss how the industry slowly came to understand the surprisingly hard problems around “flexible media,” what’s been done to solve them, and the pain points and open questions that remain.
Eric Portis: One of the interesting points I’ve heard you make — which is a nice entry point into all of this — is that images on the web have always been complicated. Can you give me a sense of what problems developers were facing in the early days of the web? What did we have to worry about then, that we don’t have to worry about now?
Jason Grigsby: So basically, you’re starting off by asking me, okay grandpa—
EP: [Laughs]
JG: —Grandpa, tell me what it was like, I’m too young to understand. I see how you are! Well, in the earliest days, the web started without any images at all, and when we first started incorporating them, the problems were tough. There were differences in color space between Macs and PCs, and we had to make sure that images were compressed well enough to download on connections that are much slower than we have now. Lynda Weinman, of Lynda.com, wrote books — I think it was five books — that were each several hundred pages long, all on images, and they were best sellers. In addition there were a number of specialized tools — I remember a piece of software called Debabelizer — that allowed us to handle images and GIF compression much better than Photoshop did. It took awhile for Photoshop to add the ability to export for the web.
Linda Wineman’s Designing Web Graphics, Coloring Web Graphics, Preparing Web Graphics, and Deconstructing Web Graphics - 1,333 pages all about images!
So yes, I think there is a knee jerk reaction that a lot of people have when they start looking at responsive images. They think that the responsive images syntax is too complex and that it should be simple like images have always been. I reject that. Images are complex by their very nature. We’ve gotten good at automating some things and we’ll get better at automating others in the future. But to think that they’re not complex is to deceive yourself. They’ve always been complex.
Even after all of the time that I’ve spent on them, I still don’t feel like I understand the full complexity of images. And I think about them far too much.
EP: Let’s fast forward to 2010 or so, when developers started asking hard questions about images in responsive layouts. You were right there at the beginning of those conversations, saying that, hey, something here is broken. What were the first problems that you and other developers ran up against? What were some of the early ideas floating around about how to solve those problems?
JG: Back in 2010, I wrote an article with the unfortunate title, “CSS Media Query for Mobile is Fool’s Gold.” I wrote it because about a year before – in 2009 – my company Cloud Four was brought in to consult on a site that was having a ton of performance problems. The site used media queries to do what we now call “responsive design,” before Ethan Marcotte coined that phrase.
We dug into what was going on, and found that the media queries were hiding assets. On this site, it was a Google map, which wasn’t necessary on small screens. But we discovered that the map was still loading – along with 47 other files, tiles and a bunch of other stuff – which made the page slow to a crawl.
After Ethan wrote “Responsive Web Design”, I started seeing similar problems everywhere. There were pages where images were hidden but still downloading in the background. Similarly, images that were far too large were being shrunk down for small screens. As a result, you’d have a page that looked great, but it was 5MB in size.
The analogy I make for this problem is that it’s akin to having a dinner party, but not having time to clean, so you shove everything into the closet. It may look like your house is clean, but you’ve really just hidden the problem and you’re ok if no one opens that door. But on a website, even if it looks good, you can’t hide the performance impact of large assets.
About a year later my co-founder Lyza Gardner and I started writing “Head First Mobile Web,” and we wanted to have a chapter on “mobile first” responsive design. At the time people thought, ok, if we start with a minimal set of small assets and only load larger assets if they’re needed, we can solve the responsive image problem. So I started looking at the techniques people that were using to do this, because I assumed that enough time had gone by — that by then somebody had to have a good solution. But I found that every single technique had problems. They were all insufficient, even on high-profile sites. The Filament Group built the Boston Globe site with assumptions based on browser behavior that they saw in 2010, when they started working on the project. But by 2011, when the site launched, that behavior had changed and their attempts at responsive images were no longer working. Experiences like that showed us that this needed to be fixed at a lower level. Things needed to change from a standards perspective.
The Filament Group’s landmark 2011 Boston Globe redesign proved that responsive design could be practical (and performant!) at scale. It also highlighted the “Responsive Images” problem – Filament Group developer Mat Marquis wrote prominently about the image problems that the Globe team encountered (and, later, founded/chaired the W3C Responsive Images Community Group, which solved them).
EP: One thing I’ve found curious is that the push for responsive images really came from the bottom up — from developers like Cloud Four and The Filament Group — rather than from the top down. And initially, there was a lot of push back from browser vendors. Do you have any insight into why this problem was more visible to, more impactful for, developers?
JG: Well, first off, not everybody was pushing back. Bruce Lawson [who works at Oprea] gets credit for coming up with the <picture> syntax. Other people were resistant, because they may not have seen the responsive design tidal wave that was coming. So you have people who aren’t really thinking about mobile yet, or who were used to sites that were using device detection to route visitors to different, device-specific pages – and they couldn’t see how important this problem was, they didn’t see it as an issue.
Plus there are some tradeoffs that happen in responsive design that we still haven’t quite resolved. By not supplying the height and width of images, we’re affecting browser’s ability to lay out pages quickly, and there’s a jumpy effect that happens when images come in. Browser makers were more attuned to those issues. And they had just finished doing a ton of performance optimization work to try to load images as quickly as possible, and, at first, it seemed like there was no way to do responsive images without breaking those optimizations.
So it wasn’t like browser vendors were saying, “we don’t want you in the clubhouse.” They really just had a different worldview. But the people who were building sites had come to realize that responsive design was inevitable and really important; that device diversity had gotten to the point where there was no point in making distinctions between a mobile site or a tablet site because some mobile devices were large and some tablets were small. But browser vendors weren’t there yet.
EP: Speaking for myself, and coming at it from the developer side, I know that I didn’t understand anything about low-level browser performance, and had no concept of how layout really worked at a deep level. I was demanding tools and not understanding their potential impact – I couldn’t see the bigger picture. And it took a while to have that conversation – for the people who make web sites and the people who make web browsers to get into each other’s heads.
JG: In May 2012 I wrote about the preparser, two years after I started looking at responsive image issues. It took me that long to get to the point where I fundamentally understood that what we were talking about was a conflict between the browser, which wants to start downloading assets as soon as possible – before it starts laying out the page – and developers working on responsive designs, who want to delay and only start downloading images after layout. Because if we wait until after layout, we can download an image that fits. That fundamental conflict is what took so long to solve, from 2010 when we started identifying these problems, until 2014 when solutions started rolling out in browsers.
EP: And full support for the responsive images spec only rolled out in the last couple of browsers – mobile and desktop Safari – this past February. That felt amazing, like — ok, finally, it took four or five years but — we did it! This is complete.
JG: You know I had to write a bio for an article that just came out in A List Apart, and I ended it with: “He never wants to talk about responsive images again—”
EP: [Laughs] Sorry.
JG:“—he never wants to talk about responsive images again, but he has ideas for two more blog posts.”
[More laughs]
JG: Which is true. I really do have ideas for two more.
Thus concludes part 1 of my three-part interview with Jason Grigsby. Stay tuned for part 2, where Jason and I dive into today’s solutions and the prickly problem of Responsive Image Breakpoints.
