Discs, filesystems and Macs – Interview with Drew Thaler

Best known for a recent series of posts on Sun’s ZFS filesystems, Drew Thaler has worked on many projects at Infinite Loop in the last decade. One of his areas of expertise is filesystems and optical discs, on which he is working right now at Sony but which also is the underlying theme on his blog, aptly titled “Recording artist”.

We contacted him to ask some questions and he very generously answered providing a lot of interesting and background info and tidbits on Apple and its technologies and inner workings.

Stories of Apple: Would you tell our readers a bit about yourself and what’s your connection with Apple?
Drew Thaler: Hi! My name is Drew Thaler, and I’m a low-level software engineer. I’ve spent most of my career working on Mac OS software, including an internship at Apple during college, and five years working at Apple.

SoA: What parts of the Mac OS did you work on?
DT: I started out in CPU Software working on drivers for new hardware: video acceleration, PCMCIA, hardware DVD playback, and more. Later, I was part of the small group of engineers that created Apple’s (Mac) OS 9 and (Mac) OS X CD and DVD burning solution, called DiscRecording.framework. And recently I spent just over a year contracting for the CoreOS filesystems group.

SoA:You also contributed to iTunes. When was that? And how was the perception at Apple of the app’s importance? Was the iPod + (Music) Store strategy already known?
DT:My friends and I developed the CD/DVD burning technology at a company outside of Apple, and we were acquired. It’s a little crazy to think about this, but iTunes had no support whatsoever for CD burning at the start of November 2000. The paperwork for the acquisition finished up in the middle of that month. Six frantic weeks of development later, we had integrated it into both iTunes and the Finder in Mac OS 9 in time for it to be shown at MacWorld San Francisco in January 2001. It shipped to customers later that month. I continued to work with the iTunes engineers until late 2003.

iTunes has always been what we call a “flagship” application for Apple. It is one of the most visible parts of the Mac OS and Steve Jobs personally oversees many of the decisions about its interface. Steve was instrumental in taking SoundJam and removing features from it to simplify, simplify, simplify. I have the utmost respect for the iTunes engineers for putting up with that, since Steve is a perfectionist with a bit of a short temper, and he’s never been easy to work with!
I’m not actually sure whether the iPod and iTunes Store were planned when iTunes was first acquired. By the time we joined up, however, iTunes was successful and the iPod project was already in motion. It was incredibly secret, even by Apple’s standards, but we still heard internal rumors about it because we were working so closely with the iTunes team. The actual name was a surprise though. I remember thinking to myself, “What kind of a stupid name is ‘iPod’?”

SoA:This “company outside Apple”. Which company was that? Can you provide any additional info?
DT:We were at Prosoft Engineering in a subsidiary called Radialogic which was about to be spun off into a separate company. Apple never actually formally acquired Radialogic, since it wasn’t a separate entity, but instead made a deal with Prosoft to acquire Radialogic’s assets and hire its engineers.
There were five of us: John Bertagnolli, Ed Wynne, Mike Shields, Nathan Duran, and myself. We’d created Radialogic’s “CD Master” and “Storage Master” products. The CD Master engine became the “Authoring Support” extension in OS9, and our FireWire and USB drivers for external CDs and DVDs became the “FireWire Authoring Support” and “USB Authoring Support” extensions.

SoA: I remember one of the initial issues of Mac OS X was disc burning. Can you provide any background on that?
DT: Yes, I do. We’d designed our CD-burning software on (Mac) OS 9. We did it with (Mac) OS X in mind, but because we did it outside of Apple, it was all theoretical — none of us actually had any free time where we could play with (Mac) OS X until, literally, the end of January 2001.
At that point we started working on a Mac OS X port. But we quickly found that it didn’t support many of the features that we needed: no direct access to the CD burner, no way to handle blank CD media, not even recursive mutexes in the pthread library! Of course we quickly started talking to the right people in CoreOS to get these problems solved, but still, there was just so much missing that we were planning on a June or July 2001 release of CD burning in Mac OS X.

Now as we were working on resolving all those problems, Mac OS X 10.0 was released. Sometime around the end of March — maybe even on launch day? — Steve Jobs announced that “CD burning will be available in Mac OS X by the end of next month.” In our group, we all freaked out. WHAT!? Nobody had told us that!
Suddenly everything changed. We went from “Let’s do this the right way” to “Do it as fast as possible, at all costs!” We hacked around the pthread problem, wrote our own quick-and-dirty kernel extension, and got it all ready to ship in literally four weeks. Mac OS X CD burning support was released on May 1st as part of 10.0.2, a day “late” but actually three full months ahead of our original plan.

SoA: You mentioned hardware DVD playback. Was your work aimed at desktops or PowerBooks? Do you know anything about Apple’s choice to not support the PowerBook G3 decoder card under Mac OS X?
DT: The DVD decoding hardware was very similar in the desktop and Powerbook. Another guy, Mike Puckett, wrote the drivers for both and I helped him maintain them. The chips were C-Cube ZiVas that did almost all of the decoding work: they just needed to be fed chunks of DVD data.
Mike gave the project the code-name Oubliette. That was a bit of dark humor because it consumed a lot of time and effort. One day I asked him, “Hey, why did you name the project Oubliette? The dictionary says it means a trap — a big hole in the ground that you fall into and you can never get out of. Oh wait. Um, never mind.”

Probably the reason Mac OS X didn’t support the decoder cards was that the Macs which had them were on the verge of obsolescence already. There was an awful lot to do in Mac OS X when it first came out and it simply wasn’t a good use of engineering resources.

SoA: In your resume you state you “developed technology demo of subpixel font smoothing on LCD screens”. Was this for Mac OS 9 or X?
DT: This was for Mac OS 9. It was 1998 and Apple had recently announced a five-year cross-licensing agreement with Microsoft, so all of their patents were fair game. CRTs were still the norm for desktop machines, but I was working in the Powerbook group where all we had were LCDs. So naturally Microsoft’s ClearType caught my eye.
I developed a small demo application which showed off the difference between whole-pixel anti-aliasing — which (Mac) OS 9 already supported — and ClearType’s sub-pixel anti-aliasing. Sub-pixel anti-aliasing looked a lot better, especially on the LCD displays we were using at the time which were lower resolution than today’s. We showed it to a few people and almost everyone who saw it seemed interested. I left Apple soon afterward to work on CD burning, so I don’t know what happened to it after that. But a few years later sub-pixel anti-aliasing showed up in Mac OS X. It was probably just that it was an obvious thing to do, but it’s nice to think that maybe my old demo had influenced someone. :-)

SoA: From your point of view how was the transition between Mac OS 9 and Mac OS X and how do you feel their coexistence was managed?
DT: It was a very difficult transition for Apple to make, and it took a very, very long time. So much was different, and a lot of things had to be completely redesigned and rewritten! After four full years of work I think they simply made the decision to push 10.0 out the door whether it was ready or not. :-)
I don’t think Mac OS X was totally ready the day it was released, but it did show a lot of potential. Once it was released it simply took some time to mature, and today I think most of us can’t imagine a world without it.

For the first year or two of Mac OS X’s existence, almost everybody at the company was working a little bit on both systems. New computers were able to boot into both Mac OS 9 and Mac OS X, so drivers had to be written for both. iTunes and our CD burning software were being released for both. And so on. Everything was literally twice the work! By the time 9 was retired, we were all more than happy to be X-only.

SoA: What do you think about Apple’s HFS+ filesystem?
DT: I think it’s a pretty great filesystem which has scaled incredibly well. HFS was first released in 1985, when the state-of-the-art was a 20 megabyte hard drive. HFS+ added a few minor changes (Unicode support, increasing the maximum file size, etc) but is essentially the same design. Journaling was a very important addition in 2003 which has extended its lifetime. These days high-end Mac OS X servers use HFS+ on 20 terabyte RAID installations. That’s a million times larger than it was originally designed for!

SdA: In the last year there’s been a strong steady buzz about Mac OS X and ZFS. What’s your take on it? And do you think Apple will (or should) actually switch from HFS+ to ZFS?
DT: I think ZFS is great. (And I wrote several blog posts recently about it: ZFS Hater Redux.
The difference between HFS+ and ZFS is very similar to the difference between Mac OS 9 and Mac OS X — it’s like the future! ZFS does so many things that HFS+ can’t: end-to-end data integrity, storage pools, snapshots, smarter caching and prefetching, sparse files, compression, I/O sorting and priority, and more.
And ZFS is quickly shaping up to become a common Unix filesystem standard: it’s open-source and many other Unix systems (Solaris, FreeBSD, NetBSD, Linux) either have added support for it or are working on adding support for it. The benefit of a shared standard is huge, because it lets everybody work together on improving the future of storage.

I think Apple will inevitably switch to ZFS over time. It won’t be a sudden change — instead it will be much more gradual, the way that HFS+ was introduced. Leopard has been widely reported to include read-only ZFS support. It won’t be long until read-write support shows up, and eventually new Macs will be able to boot from ZFS. In two to three years — 2009 or 2010 — we will probably see the first Macs which ship with ZFS as the default filesystem.

SoA: One of the most missed features of Mac OS X, filesystem-wise, is not having full and official support for Linux’s ext3 and Microsoft’s NTFS. What’s your take on this and -if you know something about it- what was or is Apple’s stance on the matter?
DT:This is pure speculation, but I think the reason Apple hasn’t officially and fully supported ext3 and NTFS is partly a matter of resources: it would be very expensive in terms of engineering time, and the return on the investment would be minimal. Most of Apple’s customers would never use either one.

NTFS is a special problem because it’s not really a published and open standard. Microsoft is free to change it at will and with no advance warning. If Apple enabled write support in NTFS, the next change Microsoft makes to it might be incompatible with Mac OS X. Can you imagine how much trouble and bad press Apple would get if OSX accidentally damaged NTFS disks that were attached?

SoA: I see you did driver work for “some of the first USB and FireWire storage drivers”. What do you think right now is the advantage of FW over USB. Is there one? Do you feel Apple is abandoning FW?
DT: There’s very little difference from the user’s perspective. They have similar speeds, both are hot-pluggable, and whatever you can do over one you can usually do over the other. USB actually has the advantage for most purposes because it’s universal in the PC market these days, has a single connector (FireWire 400 has a different connector than FireWire 800), and is a tiny bit cheaper for the manufacturer.

The primary difference between the two is that USB is host-driven, while FireWire is peer-to-peer. In a USB network all communication is routed through the host computer, much like the Stories of Apple web server is the central point which connects the authors and readers of the site. But in a FireWire network any device can talk directly to any other device — a computer doesn’t even need to be present. So, for example, a video camera can talk directly to a hard drive without a computer in the way. FireWire’s peer-to-peer design is more powerful, but also much more complicated. For whatever reason the trend seems to be that high-end audio-visual components have adopted FireWire, while most low-end computer peripherals have adopted USB.

SoA: How would you sum up the years you spent working for Apple?
DT: Fun and sometimes frustrating. Fun because there are a lot of brilliant people that work there and you get a chance to create some great technology. But it can be frustrating because there are a lot of internal politics that aren’t normally visible to the outside world.
Do you know the saying that a corporation often takes on some of the personality of its CEO? Like Steve Jobs, Apple is brilliant. But its management structure is very power-focused and hierarchical, and often paranoid and secretive even with its own employees. You don’t always feel like everybody is playing on the same team. That makes it a challenging environment to thrive in. I think it’s a little bit like going “behind the scenes” at Disney (or Edenlandia, or wherever): a little bit of the magic is gone when you know how it all works. :-)

SoA: What are you doing now? Is it related to what you did at and for Apple?
DT: I’m working at Sony Computer Entertainment America (SCEA) on a filesystem for PlayStation 3 games. The work I do is mostly focused on making games load faster from the Blu-Ray disc: compression, I/O sorting, and hard-disk caching are just a few examples. In a way it’s the reverse of what I did at Apple: with DiscRecording I worked to get bits onto optical discs, and now I work to get them off.