We’re supposed to believe that this overwhelming traffic load was unprecedented and brought their systems to a screeching halt.  Well, at least AT&T’s systems — Apple’s systems seemed fine if you weren’t going through the eligibility portion of the check.

Here’s the problem, though — if you run the numbers, and know something about web/database applications, it just doesn’t add up.

13 million database queries sounds like a lot.  But let’s say that all of these queries largely happened in the first 12 hours of the day yesterday, instead of spreading them out over the full 24 hour cycle.  That’s 1.08MM queries per hour, or 300 queries per second, on average.

I don’t know if it sounds like a lot to you, but it’s really not.  Here’s a Google query on “mysql queries per second” just to get a general idea of what people are doing out there.  Many of the results range from 2003 through the present, and folks are doing a LOT more than this.  With clustering and various attempts to scale out, folks are doing 10-20K per second.  Oracle, properly tuned, can do thousands to tens of thousands of transactions (operations that change data, not just read it) per second.

I’m not a database expert, but I’ve worked around and with them for years, and I’ll say that 300 queries per second on average is not something that should cause one of the largest (and oldest, if one considers them the heir of the Bell System) telecom companies in the world to crumple under the load.

But traffic is bursty, not uniformly distributed.  So even if they saw periods with 10-50x greater load than average, we’re still in the ballpark for reasonable performance on a pure database query.  Note that I’m assuming that eligibility is a somewhat simple database query; we gave three items of data which obviously form a compound primary key, and AT&T is supposed to return some information about eligibility for upgrade:  perhaps date, perhaps a few other bits of info.

Let’s be generous and assume that 1K of data per eligibility request is returned (i.e., there’s little concern for efficiency).  That’s still only about 300K bytes per second of query results flowing back to Apple from AT&T, or about 2.4Mbps.  Again, perhaps bursting to 20-100Mbps for very brief periods of time.  In other words, a couple of DS3s or a fast ethernet cross-connect are sufficient to carry the data back and forth.  One imagines this shouldn’t strain AT&T’s internal network too much, despite random claims yesterday.

Of course, maybe the problem here isn’t database performance or bandwidth, but that AT&T did the eligibility checks as API calls through a large enterprise system where a single check builds and then tears down many EJBs or other enterprise objects. This might be closer to the truth for a performance bottleneck here.  Maybe the system was built to handle tens, but not hundreds or thousands, of requests per second.  That’s plausible, but kind of stupid for a large engineering company used to having millions of subscribers and doing business globally.  But I could buy it.

But you’d imagine that they’d have learned something from three previous “major” iPhone releases, and the iPad 3G release, and figured out an easier way to quickly respond to eligibility requests.  After all, my eligibility isn’t a rapidly changing variable — I’m eligible on a certain day, and they know what that day is.  Which means that the eligibility of every iPhone owner on the planet could have been precalculated easily just before the iPhone4 launch, and cached.  It’s not that much data, frankly.  You could have cached a table with the user’s phone number, last 4 SSN, and zip (the keys they ask you to enter) hashed, and a eligibility “price code”, in a few gigs of memory on all the app servers, and just statically responded to queries for the first 24 hours, if you were worried that your enterprise systems wouldn’t handle “first day” load.

Anyhow, these are just ballpark figures, and they could be wildly wrong about the instantaneous loads experienced, etc.  But the general point is, 13MM eligibility checks and 600K preorders isn’t really a lot of load and traffic.  Ask Amazon or eBay what “a lot” of transactions looks like.

Or better yet, AT&T, before the next launch, hire some of their ex-employees to take a look at your databases and systems.  Please.

But one exchange is worth analysis and our attention, whatever the details of the device and our first impressions. Cory Doctorow, open-source freedom fighter extraordinaire and speculative fiction author, published a widely discussed, negative essay concerning the very idea of the iPad. By now, you’ve probably read it, or seen the link. If you haven’t, you should.

Cory’s essential points are two (with apologies if I’m missing something serious). First, that open platforms (think Linux, Android, FreeBSD, etc) are structurally designed to foster innovation at minimal entry cost, and with minimum friction to the innovator, and minimal interference between the innovator and the eventual consumer of those innovations. Second, Doctorow argues that the justification everyone is citing for the closed system — “making computers easy for mainstream users” — is insulting to mainstream users.

Joel Johnson responds that Doctorow’s principal arguments miss the point. In particular, that openness and innovation are not causally linked to the extent that open-source and Linux advocates claim. That innovation will thrive on the “nearly closed” platforms like the iPad and iPhone.

The argument is one that technologists and designers ought to think about pretty deeply. In fact, whichever side of the debate you tend to come down on, I believe you ought to sit and think hard about reasons why the other side might be right. Because I believe something like the sum of Doctorow and Johnson’s arguments represent the future of computing.

It’s undeniable that truly open systems (as opposed to the controlled openness of late 1980’s “open systems”) like Linux have fostered a Cambrian explosion of innovation and tinkering. Many of us, including myself, got their start in the computing world just as the Internet took off as a public phenomenon, with Linux, Apache, and Netscape as the principal “rocket boosters” for its success. But there are signs that the Cambrian explosion has begun to slow from a proliferative to a winnowing phase, at least in some areas.

There is still huge innovation on the server side, with new server-side frameworks and web toolkits, even new languages (i.e., the close coupling between the origins of Lift and Scala) and computing paradigms (clouds, Hadoop distributed computation) appearing monthly. When one examines the server and deep application side, it’s hard to imagine that Doctorow isn’t exactly right. Preservation of the openness of systems, the malleability of interfaces and even programming paradigms, is crucial to keeping the recombination going in this server-side cauldron of computing DNA.

Winnowing is occurring on the user experience front, however. The winnowing-out process really began when we all decided that the web browser was the main arbiter of client-side experience. In fact, that seems so obvious today that we forget that the browser used to simply be an HTML display engine, not a general-purpose UI framework for multi-language code execution. We forget things like Pointcast, and other “rich internet client” examples, since for years they largely went away in favor of browser-based apps. Even in “thick” applications like Microsoft Office, when internet interaction was required — say, for Help or looking up a template, the thick app would spawn a little unmarked IE window to do the “internet stuff.”

Johnson’s point, as I take it, is that this winnowing process on the user experience front has been a good thing, and should continue. And I agree. And Doctorow is wrong to focus on the “making a computer my mother can use” trope. The intuition behind the trope is fine, as far as it goes, but the metaphor fails to dig down to the core of the issue. Certainly, we nerds have done an insufficient job of hiding the underlying execution model of our software, and a poor job of interface and experience design, by and large. Despite the efforts of Don Norman, Alan Cooper, and others, we in the open systems community continue to focus on the computational underbelly and not enough on the user’s experience.

Which is where Apple comes in. We all know that the reason their devices continue to delight and appeal is that they are designed in every detail, by teams of design fanatics, led by our industry’s chief design obsessive. And we all tell ourselves that the degree of design fanaticism that continues to suck us in is only possible with closed or semi-closed systems.

And this is where we find ourselves today. Those, like Doctorow, who value open systems for their innovation- and freedom-enhancing qualities, steer clear of Apple devices. Those, like Johnson, who value the progress we’ve made toward highly usable, well-designed systems, eagerly await new Apple designs.

I’d like to suggest that the distinction isn’t as sharp as we think, however. The design fanaticism displayed by Apple is not inherently a feature of their approach to closed systems; closed systems simply make it easier. The closed system is really just a choice, not a necessity, if one wants design excellence. Similarly, the lack of design excellence displayed in most open-source software (I’m thinking about Linux desktop apps in particular here), is not inherently a feature of an open-source community. Open-source projects simply make it harder to create and enforce design excellence.

One of Johnson’s points, as I read him, is that if we want innovation in applications, coupled with performance and design excellence, we need to stop reinventing the wheel on everything else constantly, and innovate at higher levels. And that’s easiest when all you have to worry about is the application you’re designing, not the platform as well.

Personally, I would love to see a world where we continue to see huge innovation on computing models in general, and free access to the fundamental tools for development, and options for retaining control over security and privacy. But in that world I’d also love reliable, gorgeous, high performance platforms like the iPad, as the mechanism by which the innovation that openness engenders is available to everyone. That’s likely possible, in practical terms, only with a mixture of openness and closed systems.

The downloading process exposes some seriously irritating bugs and/or design flaws in iTunes, however. I live at the northern edge of civilization on an island (well, my Canadian friends would say the southern edge, and after reading coverage of the Tea Party Convention I’m inclined to agree…) and I have “difficult” internet connectivity. This is no fault of my local ISP, who do an amazing job considering where I live.

But I often encounter TCP resets in long downloads given the Motorola Canopy point-to-point wireless I use, and iTunes really behaves badly. Despite having typed my Store password to begin the download, upon resumption, iTunes will ask me again. And again. And again. Possibly once for every stream that needs to be resumed, but it doesn’t seem to be as well patterned as that. The application hasn’t restarted, I haven’t logged out, it’s the same hardware underneath, why can’t the application cache the Store password used to initiate a given set of downloads for the duration? Perhaps only asking me to retype if the application closes and restarts?

This seems trivial, but if it happens frequently, and you’re not sitting in front of the computer to type your password whenever needed, downloading a season of episodes can literally take days. Three thus far, in fact, for a show I’m subscribing to at the moment. With 29 more items to go. Basically, it’s going to take a week of retyping my iTunes Store password to get the entire season down, given my internet connection (which is normally pretty decent for browsing and other purposes).

Doesn’t anybody in Cupertino test this type of use case?

Of course I recognized the name, and that I’d been a user since their beta release, and I remembered renewing my license again this year, but what I couldn’t immediately remember was whether that software was still an integral part of keeping my information current, sync’d, backed up, etc. Basically, is it necessary, or is it cruft?

That’s a general problem these days, and arguably it’s a worse problem on the Mac platform than on Windows, though of course it exists there as well. It’s more of a problem because Microsoft tries to build more of this stuff into Windows itself and its major desktop/server suites. Apple leaves more of it to the ISV community.

And as I noted in a previous post, good Mac software can be had for twenty, forty or sixty bucks. So people, especially professionals and developers, have a tendency to buy new apps just to see if it’s a bit better than the previous generation. I’ve done that with notetaking software, outliners, todo list management, and a bewildering variety of synchronization, backup, and storage apps and utilities.

All of which means that my laptop consistently has more than one “appendix” running — part of the system but functionally useless because it’s not being used.

And all which contributes to complexity and difficulty in troubleshooting. When my contacts database suddenly is empty, or has three or four copies of every contact (both of which seem to happen to me), which link in the synchronization chain is responsible? Is it syncing Address Book to Google Contacts? Plaxo syncing with Address Book?

Ultimately, to manage all this complexity, we’re going to need to be able to map the information flow between applications, so I can ask the question and get an answer. Today, I have to sit down and check each app’s preferences and configuration, and sort of make a list of where things are flowing, and rebuild the picture every time something goes wrong.

In complex systems, just as much vital information is contained in the links between things, as in the things themselves…

There’s a lot of criticism about what the device doesn’t have built-in, or doesn’t support. And there’s been a lot of “why, it’s nothing but a big iPod Touch.” And the usual lists of “must have but missing” features from engineers and developers who are already gnashing their teeth about how useless the iPad will be.

Here’s why we should ignore premature predictions of doom for the iPad.

Sure, there’s nothing shockingly new here. In a sense, it’s a big iPod Touch. Or it’s a slimmed down Tablet PC with integral Kindle. Actually, it’s all of those things.

What we’re forgetting is that Apple’s main strength isn’t necessarily inventing a new category (marketing spin aside), it is in bringing hard-core user research and industrial design to bear on creating devices which end up “crossing the chasm” to the mainstream for a given technology. THAT is what Apple, and Steve Jobs, are good at. I know it’s hard to remember this far back, but in the late 1990’s many of us had MP3 player devices. I had a big clunky one from Creative Labs, that was crafted to look exactly like an old-school Sony Walkman CD player – despite the fact that I was playing MP3 files, somehow it seemed like a good design decision to make the player flat and round and consequently bulky. MP3 players existed, but let’s face it, in 1998 the “mainstream” didn’t have them — your grandparents didn’t have them to take golfing, or walking the beach, etc.

Apple changed that with the iPod. And it wasn’t just the device, it was the ecosystem, and the support, and the deals they made with record labels who were in the midst of watching digital music eat their business.

We also had smartphones. If you’re like me, you probably had several generations of smartphones before you touched an iPhone. I had several Palm phones, a Treo running Palm, a Treo running Windows Mobile, a Blackberry or two, etc. With the exception of Blackberry in business and government markets, smartphones became a “mainstream” phenomenon with the iPhone.

“Tablets” in various form factors have been around awhile too. There are the modified laptops like the Toshibas. I had one at Microsoft, and it was a “tablet” only in the sense that the screen turned around, rendering the entire 6 pound, 2 inch thick laptop a bulky version of a paper writing tablet. Personally, I think the guys at Motion Computing have been the “tablet” makers to watch — thin, small, and I lusted after one apart from the fact that it ran Windows and was underpowered to do so.

So what I think we’re going to see with Apple and the iPad is that they’re taking the best of the tablet PC tradition — i.e., devices like Motion Computing — book readers like the Kindle, and the app ecosystem of the already successful iPhone, and blending it with their unerring ability to do solid industrial design for high technology.

I have no idea what applications the iPad will find. Will it become big in health care, where Motion Computing has made inroads? Will it replace PCs at home for many people with the need only to do email, browse the web, look up information, and manage photos? Who knows at this point, really.

But if you’re betting the iPad will be a failure at this point, just from looking at the specs and what components it does or doesn’t contain, you’re ignoring the big picture. Which is that Apple has an absolutely stellar track record of looking at developing technologies and areas of application, pushing their engineers and designers to produce something easy to use and gorgeous to look at, and then marketing it relentlessly to large, mainstream audiences.

That’s not a guarantee of success every time. But if Jimmy the Greek were still with us, he wouldn’t give you good odds betting against Apple on this one.

It’s game changing not for third-party ISVs already developing for the iPhone, since they’re used to charging 99 cents to a few bucks for an app. For Mac software developers like OmniGroup, it’ll be challenging. There is already a large Mac software ecosystem with apps priced in the $20 – $60 range. These ISV’s have continued to charge such prices even while iPhone app prices dropped a zero, because the difference in functionality and screen size between a Mac laptop and the iPhone is significant. The difference in what users can do is significant.

iWork on the iPad is a laptop/desktop experience, suitable for the vast majority of home and many business users. And yet Apple dropped a zero on the pricing, basically. With a presentation program, word processor, and spreadsheet available for $10 each, or $30 for the entire productivity suite, how will third party ISV’s charge $50 or $60 for an iPad version of their Mac software apps? Perhaps they can’t.

The problem, of course, comes when iPad gains measurable market share, not when rabid Apple fans line up for first-day sales. Give it a couple of years, and there’s a couple of million of these things floating around, and ordinary people buy fewer laptops and have iPads at home or for travel instead. They’ll get used to buying iPad apps for a few dollars more than a pure iPhone app. They’ll get used to being able to do 75, 80, or even 100% of what they used to do on their laptops or desktops (again, I’m talking about non-developers, non-IT professionals here).

And they’ll start rebelling against the notion that a multi-touch capable, gesture controlled, “natural” feeling user experience should cost $10 or $20, but when they need to sit down at a laptop or desktop computer and go back to keyboard and mouse, the OLD experience should cost $50, $100, or more.

So third-party ISVs should be preparing for another phase transition in software pricing, downward. As always, our demands for functionality and usability and seamless integration rise, and our tolerance for premium pricing drops.

But really who’s in trouble given this pricing is Microsoft. Despite iWork on the Mac, the reality is that Microsoft Office still has a lock on the productivity tools market. Especially in businesses. That hasn’t changed, and it won’t change tomorrow.

Nevertheless, anyone who thinks that Microsoft can continue to defend a price differential of hundreds of dollars for Office apps vs. iWork on the iPad, once the iPad gains market share, isn’t paying attention. Businesses, especially US-based ones, are increasingly challenged to control costs and compete in a tough economy.

Office Standard 2007 costs $400, but let’s say you upgraded from a previous version. And let’s say you’re a small to medium business, and for round numbers you’ve got 100 employees. To upgrade everyone to Office Standard 2007 costs you $23995. To instead purchase an application suite priced like iWork — $30 for the bundle, is $3000 for everyone in your company. You just saved $21,000.

Leave aside the details – whether desktop software really will drop to the iPad level. Whether the ease of interoperating with Office is such that businesses could afford to not have Microsoft Office on their desk. The latter is just a matter of development, and the arms race to break compatibility if you’re Microsoft and recreate it fast enough if you’re a third party ISV.

But one thing is clear. A year from now or five years from now, the combination of Apple and Google are aiming squarely to cut Microsoft’s desktop software business off at the knees. Who knows whether they succeed, but the ancillary effect will be a major restructuring of the economics of rich desktop software businesses, since they live in fitness landscape created by the interplay between these large players.

And significant as the iPad hardware might be (more on this in future posts), Apple just shifted the fitness landscape with a bold move on the software side of the industry, and we shouldn’t lose sight of that in the midst of discussing lacking USB ports and cameras and so on.

This morning, fingers crossed, we’ll learn more about the new “tablet” device.  The leaks have been accelerating for days, business partners ringing my iPhone constantly to tell me breaking news, and of course I’ve read all the non-news news purporting to describe authoritative leaks.

But none of it matters, because ultimately what we want to see is Steve, dressed in his usual black and white, stand onstage and give The Demo.  If you’re in the biz, The Demo is King.  The Demo is where you set expectations, destroy preconceived notions.  The Demo is where you win or lose, fundamentally.  Because before The Demo, the chessboard is empty.  The Demo is where you put your pieces down — not in the starting configuration, but hopefully in position to reach mate in the fewest moves possible.

If you’re Doug Englebart, giving the mother of all demos, you literally change the world by showing us the ragged bits that the rest of us will spend the next forty years making smooth and usable and real.  Everything that followed:  Dan Bricklin’s Visicalc, Alan Kay’s pioneering work, Steve and Steve with the Apple II, Bill Gates, Tim Berners-Lee and the Web, Netscape, Linus Torvalds…all of it the work of giants in our field…all of it playing out the possibilities inherent in that mother of all demos.

Steve Jobs consciously aims at game-changing demos:  the original iPhone demo was, as was OS X and the Intel transition.  I don’t know that today’s announcement will rise to that level, but I hope so.

I think our industry is getting tired of playing out the possibilities inherent in a forty-year-old demo.  It would be nice to have some new territory to explore.

Last Friday night, I open an email from Amazon, and it contains a friendly reminder about my Kindle warranty and what it provides me. And in the middle, a little paragraph precisely describing what can happen if you open the default cover/case backwards — describing the cracking I’ve got. And the email encourages you to get in touch with Support.

After decades of purchasing high-tech gadgets and software, naturally I’m skeptical, but this is Amazon, so I shoot them an email and say I’d love to get the cracks fixed so they don’t get worse, but don’t want to be without the Kindle for very long.

Within 15 minutes I get a simple, auto-generated email saying that a replacement Kindle DX is in the shipping queue, and that when it arrives I should simply transfer any non-Amazon content I’ve loaded onto my old one, swap devices, and send the old one back with the pre-paid shipping label provided. In fact, I have 30 days to migrate to the new device and send the old one back (if I don’t, they’ll simply charge me the cost of the device).

I shouldn’t be blown away by this, because Amazon bases their business on fast service, shipment, resolution of problems, etc. When they estimate the delivery date of a book, I know it’s always that date — or earlier. Returns of erroneous shipments or broken items have always been easy. So I should have expected this kind of treatment for the Kindle.

But I didn’t. Somehow, when the product or service is highly technical — not just an order of toiletries, CDs or books — I’ve come to expect crappy service and support. Haven’t you?

Amazon (and to a significant, but lesser extent Apple) continue to show that customer service and advanced technology and services can go together quite nicely. Kudos, Amazon. And thanks — not just for fixing my Kindle, but going out of your way to remind me that I had every right to have my cracked Kindle fixed.

Back in 2003 on my "previous" blog, and in early 2005 on this blog, I updated a long-standing essay I’d
called "A Personal History of Personal Computing." My first and second blogs are long
gone in the transition away from Radio Userland to Typepad, but I think
it’s time to reprint and update that essay (a second time). Moore’s law is one way to
look at the history of personal computing. Another is the history of
companies that have come and gone, making personal computers and
software. Still another is a personal view. This story is about my own
personal computing history — the machines, what I did with them, what
software I thought was important. I omit computers that I didn’t really
have control over, such as University mainframes and Unix servers, and
I also omit the vast array of servers and computers I administered at
RealNetworks, Internap, Network Clarity, and computers I used at Microsoft and now GridNetworks.

By my count, I’ve purchased 21 computers in my life, and of course used and worked with hundreds, if not thousands more (managing a Systems Engineering group will do that for you). 

The story starts in the late 1970’s, shortly after personal computers came about and before IBM changed things forever…. 

It’s a simple thing these days to do just-in-time manufacturing and shipping from China, and I wasn’t expecting my Macbook Air until Feb. 12. But Apple pulled the shipping date in by a week, and was going to deliver it on Feb. 6th. I was thrilled.

It arrived this morning. Sure, it’s not that complex, but boy, do they know how to make a tech geek happy. The packaging is gorgeous — a coworker said it reminded him of a Tiffany’s box. Even the Apple skeptics in the office — the dyed-in-the-wool, live-in-Redmond-even-though-they-don’t-work-at-Microsoft types, were drooling just a little bit. When they thought I wasn’t looking. They know who they are, and today I can see that their snide comments about Apple and the Cult of Steve are just envy wrapped in sarcasm.

Then you unpack it, and the Air feels both lighter and more substantial than you expect. The screen is terrific, the keyboard very nice, and the overall experience is exactly what I hoped a subnotebook from Apple would be. Even the “Remote CD/DVD” thing works perfectly for installing software — although I bought an external Superdrive, mostly because the remote thing doesn’t work well for playing DVD’s due to the copy protection schemes.

I’m sure after a couple of days or a week of using the machine daily I’ll have the usual list of gripes, wishes, etc. But not today. Today I’m sitting on the sofa, having loaded LaTeX, Office 2008, and a few other essentials, and just enjoying that “first day” experience.

Wow. Bravo Apple.