We’re supposed to believe that this overwhelming traffic load was unprecedented and brought their systems to a screeching halt.  Well, at least AT&T’s systems — Apple’s systems seemed fine if you weren’t going through the eligibility portion of the check.

Here’s the problem, though — if you run the numbers, and know something about web/database applications, it just doesn’t add up.

13 million database queries sounds like a lot.  But let’s say that all of these queries largely happened in the first 12 hours of the day yesterday, instead of spreading them out over the full 24 hour cycle.  That’s 1.08MM queries per hour, or 300 queries per second, on average.

I don’t know if it sounds like a lot to you, but it’s really not.  Here’s a Google query on “mysql queries per second” just to get a general idea of what people are doing out there.  Many of the results range from 2003 through the present, and folks are doing a LOT more than this.  With clustering and various attempts to scale out, folks are doing 10-20K per second.  Oracle, properly tuned, can do thousands to tens of thousands of transactions (operations that change data, not just read it) per second.

I’m not a database expert, but I’ve worked around and with them for years, and I’ll say that 300 queries per second on average is not something that should cause one of the largest (and oldest, if one considers them the heir of the Bell System) telecom companies in the world to crumple under the load.

But traffic is bursty, not uniformly distributed.  So even if they saw periods with 10-50x greater load than average, we’re still in the ballpark for reasonable performance on a pure database query.  Note that I’m assuming that eligibility is a somewhat simple database query; we gave three items of data which obviously form a compound primary key, and AT&T is supposed to return some information about eligibility for upgrade:  perhaps date, perhaps a few other bits of info.

Let’s be generous and assume that 1K of data per eligibility request is returned (i.e., there’s little concern for efficiency).  That’s still only about 300K bytes per second of query results flowing back to Apple from AT&T, or about 2.4Mbps.  Again, perhaps bursting to 20-100Mbps for very brief periods of time.  In other words, a couple of DS3s or a fast ethernet cross-connect are sufficient to carry the data back and forth.  One imagines this shouldn’t strain AT&T’s internal network too much, despite random claims yesterday.

Of course, maybe the problem here isn’t database performance or bandwidth, but that AT&T did the eligibility checks as API calls through a large enterprise system where a single check builds and then tears down many EJBs or other enterprise objects. This might be closer to the truth for a performance bottleneck here.  Maybe the system was built to handle tens, but not hundreds or thousands, of requests per second.  That’s plausible, but kind of stupid for a large engineering company used to having millions of subscribers and doing business globally.  But I could buy it.

But you’d imagine that they’d have learned something from three previous “major” iPhone releases, and the iPad 3G release, and figured out an easier way to quickly respond to eligibility requests.  After all, my eligibility isn’t a rapidly changing variable — I’m eligible on a certain day, and they know what that day is.  Which means that the eligibility of every iPhone owner on the planet could have been precalculated easily just before the iPhone4 launch, and cached.  It’s not that much data, frankly.  You could have cached a table with the user’s phone number, last 4 SSN, and zip (the keys they ask you to enter) hashed, and a eligibility “price code”, in a few gigs of memory on all the app servers, and just statically responded to queries for the first 24 hours, if you were worried that your enterprise systems wouldn’t handle “first day” load.

Anyhow, these are just ballpark figures, and they could be wildly wrong about the instantaneous loads experienced, etc.  But the general point is, 13MM eligibility checks and 600K preorders isn’t really a lot of load and traffic.  Ask Amazon or eBay what “a lot” of transactions looks like.

Or better yet, AT&T, before the next launch, hire some of their ex-employees to take a look at your databases and systems.  Please.

But one exchange is worth analysis and our attention, whatever the details of the device and our first impressions. Cory Doctorow, open-source freedom fighter extraordinaire and speculative fiction author, published a widely discussed, negative essay concerning the very idea of the iPad. By now, you’ve probably read it, or seen the link. If you haven’t, you should.

Cory’s essential points are two (with apologies if I’m missing something serious). First, that open platforms (think Linux, Android, FreeBSD, etc) are structurally designed to foster innovation at minimal entry cost, and with minimum friction to the innovator, and minimal interference between the innovator and the eventual consumer of those innovations. Second, Doctorow argues that the justification everyone is citing for the closed system — “making computers easy for mainstream users” — is insulting to mainstream users.

Joel Johnson responds that Doctorow’s principal arguments miss the point. In particular, that openness and innovation are not causally linked to the extent that open-source and Linux advocates claim. That innovation will thrive on the “nearly closed” platforms like the iPad and iPhone.

The argument is one that technologists and designers ought to think about pretty deeply. In fact, whichever side of the debate you tend to come down on, I believe you ought to sit and think hard about reasons why the other side might be right. Because I believe something like the sum of Doctorow and Johnson’s arguments represent the future of computing.

It’s undeniable that truly open systems (as opposed to the controlled openness of late 1980’s “open systems”) like Linux have fostered a Cambrian explosion of innovation and tinkering. Many of us, including myself, got their start in the computing world just as the Internet took off as a public phenomenon, with Linux, Apache, and Netscape as the principal “rocket boosters” for its success. But there are signs that the Cambrian explosion has begun to slow from a proliferative to a winnowing phase, at least in some areas.

There is still huge innovation on the server side, with new server-side frameworks and web toolkits, even new languages (i.e., the close coupling between the origins of Lift and Scala) and computing paradigms (clouds, Hadoop distributed computation) appearing monthly. When one examines the server and deep application side, it’s hard to imagine that Doctorow isn’t exactly right. Preservation of the openness of systems, the malleability of interfaces and even programming paradigms, is crucial to keeping the recombination going in this server-side cauldron of computing DNA.

Winnowing is occurring on the user experience front, however. The winnowing-out process really began when we all decided that the web browser was the main arbiter of client-side experience. In fact, that seems so obvious today that we forget that the browser used to simply be an HTML display engine, not a general-purpose UI framework for multi-language code execution. We forget things like Pointcast, and other “rich internet client” examples, since for years they largely went away in favor of browser-based apps. Even in “thick” applications like Microsoft Office, when internet interaction was required — say, for Help or looking up a template, the thick app would spawn a little unmarked IE window to do the “internet stuff.”

Johnson’s point, as I take it, is that this winnowing process on the user experience front has been a good thing, and should continue. And I agree. And Doctorow is wrong to focus on the “making a computer my mother can use” trope. The intuition behind the trope is fine, as far as it goes, but the metaphor fails to dig down to the core of the issue. Certainly, we nerds have done an insufficient job of hiding the underlying execution model of our software, and a poor job of interface and experience design, by and large. Despite the efforts of Don Norman, Alan Cooper, and others, we in the open systems community continue to focus on the computational underbelly and not enough on the user’s experience.

Which is where Apple comes in. We all know that the reason their devices continue to delight and appeal is that they are designed in every detail, by teams of design fanatics, led by our industry’s chief design obsessive. And we all tell ourselves that the degree of design fanaticism that continues to suck us in is only possible with closed or semi-closed systems.

And this is where we find ourselves today. Those, like Doctorow, who value open systems for their innovation- and freedom-enhancing qualities, steer clear of Apple devices. Those, like Johnson, who value the progress we’ve made toward highly usable, well-designed systems, eagerly await new Apple designs.

I’d like to suggest that the distinction isn’t as sharp as we think, however. The design fanaticism displayed by Apple is not inherently a feature of their approach to closed systems; closed systems simply make it easier. The closed system is really just a choice, not a necessity, if one wants design excellence. Similarly, the lack of design excellence displayed in most open-source software (I’m thinking about Linux desktop apps in particular here), is not inherently a feature of an open-source community. Open-source projects simply make it harder to create and enforce design excellence.

One of Johnson’s points, as I read him, is that if we want innovation in applications, coupled with performance and design excellence, we need to stop reinventing the wheel on everything else constantly, and innovate at higher levels. And that’s easiest when all you have to worry about is the application you’re designing, not the platform as well.

Personally, I would love to see a world where we continue to see huge innovation on computing models in general, and free access to the fundamental tools for development, and options for retaining control over security and privacy. But in that world I’d also love reliable, gorgeous, high performance platforms like the iPad, as the mechanism by which the innovation that openness engenders is available to everyone. That’s likely possible, in practical terms, only with a mixture of openness and closed systems.

There’s a lot of criticism about what the device doesn’t have built-in, or doesn’t support. And there’s been a lot of “why, it’s nothing but a big iPod Touch.” And the usual lists of “must have but missing” features from engineers and developers who are already gnashing their teeth about how useless the iPad will be.

Here’s why we should ignore premature predictions of doom for the iPad.

Sure, there’s nothing shockingly new here. In a sense, it’s a big iPod Touch. Or it’s a slimmed down Tablet PC with integral Kindle. Actually, it’s all of those things.

What we’re forgetting is that Apple’s main strength isn’t necessarily inventing a new category (marketing spin aside), it is in bringing hard-core user research and industrial design to bear on creating devices which end up “crossing the chasm” to the mainstream for a given technology. THAT is what Apple, and Steve Jobs, are good at. I know it’s hard to remember this far back, but in the late 1990’s many of us had MP3 player devices. I had a big clunky one from Creative Labs, that was crafted to look exactly like an old-school Sony Walkman CD player – despite the fact that I was playing MP3 files, somehow it seemed like a good design decision to make the player flat and round and consequently bulky. MP3 players existed, but let’s face it, in 1998 the “mainstream” didn’t have them — your grandparents didn’t have them to take golfing, or walking the beach, etc.

Apple changed that with the iPod. And it wasn’t just the device, it was the ecosystem, and the support, and the deals they made with record labels who were in the midst of watching digital music eat their business.

We also had smartphones. If you’re like me, you probably had several generations of smartphones before you touched an iPhone. I had several Palm phones, a Treo running Palm, a Treo running Windows Mobile, a Blackberry or two, etc. With the exception of Blackberry in business and government markets, smartphones became a “mainstream” phenomenon with the iPhone.

“Tablets” in various form factors have been around awhile too. There are the modified laptops like the Toshibas. I had one at Microsoft, and it was a “tablet” only in the sense that the screen turned around, rendering the entire 6 pound, 2 inch thick laptop a bulky version of a paper writing tablet. Personally, I think the guys at Motion Computing have been the “tablet” makers to watch — thin, small, and I lusted after one apart from the fact that it ran Windows and was underpowered to do so.

So what I think we’re going to see with Apple and the iPad is that they’re taking the best of the tablet PC tradition — i.e., devices like Motion Computing — book readers like the Kindle, and the app ecosystem of the already successful iPhone, and blending it with their unerring ability to do solid industrial design for high technology.

I have no idea what applications the iPad will find. Will it become big in health care, where Motion Computing has made inroads? Will it replace PCs at home for many people with the need only to do email, browse the web, look up information, and manage photos? Who knows at this point, really.

But if you’re betting the iPad will be a failure at this point, just from looking at the specs and what components it does or doesn’t contain, you’re ignoring the big picture. Which is that Apple has an absolutely stellar track record of looking at developing technologies and areas of application, pushing their engineers and designers to produce something easy to use and gorgeous to look at, and then marketing it relentlessly to large, mainstream audiences.

That’s not a guarantee of success every time. But if Jimmy the Greek were still with us, he wouldn’t give you good odds betting against Apple on this one.

It’s game changing not for third-party ISVs already developing for the iPhone, since they’re used to charging 99 cents to a few bucks for an app. For Mac software developers like OmniGroup, it’ll be challenging. There is already a large Mac software ecosystem with apps priced in the $20 – $60 range. These ISV’s have continued to charge such prices even while iPhone app prices dropped a zero, because the difference in functionality and screen size between a Mac laptop and the iPhone is significant. The difference in what users can do is significant.

iWork on the iPad is a laptop/desktop experience, suitable for the vast majority of home and many business users. And yet Apple dropped a zero on the pricing, basically. With a presentation program, word processor, and spreadsheet available for $10 each, or $30 for the entire productivity suite, how will third party ISV’s charge $50 or $60 for an iPad version of their Mac software apps? Perhaps they can’t.

The problem, of course, comes when iPad gains measurable market share, not when rabid Apple fans line up for first-day sales. Give it a couple of years, and there’s a couple of million of these things floating around, and ordinary people buy fewer laptops and have iPads at home or for travel instead. They’ll get used to buying iPad apps for a few dollars more than a pure iPhone app. They’ll get used to being able to do 75, 80, or even 100% of what they used to do on their laptops or desktops (again, I’m talking about non-developers, non-IT professionals here).

And they’ll start rebelling against the notion that a multi-touch capable, gesture controlled, “natural” feeling user experience should cost $10 or $20, but when they need to sit down at a laptop or desktop computer and go back to keyboard and mouse, the OLD experience should cost $50, $100, or more.

So third-party ISVs should be preparing for another phase transition in software pricing, downward. As always, our demands for functionality and usability and seamless integration rise, and our tolerance for premium pricing drops.

But really who’s in trouble given this pricing is Microsoft. Despite iWork on the Mac, the reality is that Microsoft Office still has a lock on the productivity tools market. Especially in businesses. That hasn’t changed, and it won’t change tomorrow.

Nevertheless, anyone who thinks that Microsoft can continue to defend a price differential of hundreds of dollars for Office apps vs. iWork on the iPad, once the iPad gains market share, isn’t paying attention. Businesses, especially US-based ones, are increasingly challenged to control costs and compete in a tough economy.

Office Standard 2007 costs $400, but let’s say you upgraded from a previous version. And let’s say you’re a small to medium business, and for round numbers you’ve got 100 employees. To upgrade everyone to Office Standard 2007 costs you $23995. To instead purchase an application suite priced like iWork — $30 for the bundle, is $3000 for everyone in your company. You just saved $21,000.

Leave aside the details – whether desktop software really will drop to the iPad level. Whether the ease of interoperating with Office is such that businesses could afford to not have Microsoft Office on their desk. The latter is just a matter of development, and the arms race to break compatibility if you’re Microsoft and recreate it fast enough if you’re a third party ISV.

But one thing is clear. A year from now or five years from now, the combination of Apple and Google are aiming squarely to cut Microsoft’s desktop software business off at the knees. Who knows whether they succeed, but the ancillary effect will be a major restructuring of the economics of rich desktop software businesses, since they live in fitness landscape created by the interplay between these large players.

And significant as the iPad hardware might be (more on this in future posts), Apple just shifted the fitness landscape with a bold move on the software side of the industry, and we shouldn’t lose sight of that in the midst of discussing lacking USB ports and cameras and so on.

This morning, fingers crossed, we’ll learn more about the new “tablet” device.  The leaks have been accelerating for days, business partners ringing my iPhone constantly to tell me breaking news, and of course I’ve read all the non-news news purporting to describe authoritative leaks.

But none of it matters, because ultimately what we want to see is Steve, dressed in his usual black and white, stand onstage and give The Demo.  If you’re in the biz, The Demo is King.  The Demo is where you set expectations, destroy preconceived notions.  The Demo is where you win or lose, fundamentally.  Because before The Demo, the chessboard is empty.  The Demo is where you put your pieces down — not in the starting configuration, but hopefully in position to reach mate in the fewest moves possible.

If you’re Doug Englebart, giving the mother of all demos, you literally change the world by showing us the ragged bits that the rest of us will spend the next forty years making smooth and usable and real.  Everything that followed:  Dan Bricklin’s Visicalc, Alan Kay’s pioneering work, Steve and Steve with the Apple II, Bill Gates, Tim Berners-Lee and the Web, Netscape, Linus Torvalds…all of it the work of giants in our field…all of it playing out the possibilities inherent in that mother of all demos.

Steve Jobs consciously aims at game-changing demos:  the original iPhone demo was, as was OS X and the Intel transition.  I don’t know that today’s announcement will rise to that level, but I hope so.

I think our industry is getting tired of playing out the possibilities inherent in a forty-year-old demo.  It would be nice to have some new territory to explore.

I’m particularly excited because I think both technologies are strong and are architecturally an excellent match. SCOM 2007 is very flexible model-based management technology, and EMC Smarts is the best of the “new generation” network management systems that inherently model root-cause and network causality. The merger of the two, along with injections of SDM/SML model technology and additional vendor-specific models written by equipment vendors will create a very strong management solution that finally will do more than “ring the alarm” when a symptom occurs. Model-based management has been in the works for years, and in our own way at Network Clarity we attempted to do MBM for network configuration and optimization, but the latter is an even harder challenge to do well in some ways.

So congratulations to Rich, Vij, Shawn, Christian, Ashvin, and others on announcing this deal and making the strategy a reality!

http://www.infoworld.com/article/07/03/27/HNmsemcnetworkmanagement_1.html

http://www.byteandswitch.com/document.asp?doc_id=120547

My friend Bill and I have a gentleman’s wager: he doesn’t believe the iPhone will be successful, and I do. Our bet was based purely on our analysis of cell phone markets and Apple’s track record of hardware and system design, but today we’ve finally got details to go on. I’m going to stick with my side of the wager, especially since there’s a dinner with good Chateauneuf-du-Pape at the end of it for me. But I wonder, after seeing the keynote details today, whether Bill will feel the same way about his half of the wager.

Wow…they just announced free Yahoo IMAP email to all iPhone customers. Not that I use Yahoo IMAP, but that’s a good thing to announce alongside the hardware. Jobs has clearly learned via the iTunes/iPod and OS X experience the importance of having an ecosystem simultaneous with hardware release is critical — a lesson that most of us believe drastically held back the Macintosh in its first ten years.

I’m not going to keep translating the great live notes from MacRumorsLive.com, but check out the transcript for yourself. All that remains between me and ordering one is details: pricing, carriers, timeline for availability. Even just looking at screenshots and the transcript, the iPhone is going to make my Treo 700w look sickly in comparison.

I have some mixed feelings about leaving — except for the commute and my overall need to reduce my stress level and improve my lifestyle I thoroughly enjoyed the projects I was working on. Neither project has been generally announced yet, so I’m not going to discuss them in any detail. I will say that I saw a different Microsoft than one might have seen several years ago. I worked on two projects which were clearly enterprise management territory for most software companies, but not (traditionally) for Microsoft. And yet, I worked with people who were intensely interested in branching out into new territory, solving problems that Microsoft in the past hadn’t thought were important for Microsoft to solve, and in general breaking any number of stereotypes about what, and more especially how, to do things.

That’s not to say that I didn’t also encounter a fair amount of inertia, and people who already know “the way” that something needed to be done, but I also found most people I talked with — folks in program management, marketing, director-level staff, GMs, all the way to my chain of VPs — were open and amenable to new arguments if supportable and workable.

Clearly, there is still a lot of “religion,” and although the devotion to “all things Windows” is understandable from the Windows platform vendor, I did find that knowledge of enterprise-level mixed IT installations was relatively limited. This is being fixed by hiring from the outside — I was a case in point actually — so I expect this situation to be very different in a year. I encountered an organization that was working fast and hard to remedy any gaps it might have in order to be seriously competitive in nearly all areas of enterprise software.

Wow. That sounds like I swallowed a bit of kool-aid myself. But I didn’t, not really. I’m typing this on my Macbook Pro, and I don’t even have a PC laptop or Windows desktop anymore — all of that is virtualized by Parallels Workstation and does everything I need. My primary platforms are Mac OS X and Linux.

So the message is, that the Microsoft I spent part of a year working at is not the Microsoft of evil fantasy. The company is as flexible and open to change as the people who now run and work at it, and to my eye that’s getting more flexible and open all the time. I suspect there’s still a bright future for the folks in Redmond. I’m grateful for my time there, and happy I got to spend at least a little time seeing it from the inside.

And I’m happy to move on, to whatever is next…

Allrecipes is a project whose history you can read (or will read) elsewhere, and I won’t rehash it except to say that my co-founder Tim Hunt did an extraordinary job over the years providing not only technical, but conceptual and business leadership — firmly understanding what the site’s users wanted (which is sometimes distinct from what the business’s actual customers want) and making sure they get it. Bill Moore, the company’s CEO, also has done an outstanding job driving our execution and helping the company capitalize on the growing return towards valuing successful internet content businesses. Bill, Tim, and the whole team continue to demonstrate community interaction creates long-lasting loyalty in a medium where “stickiness” is usually measured in seconds or minutes, not months or years.

As you can read in the various press releases and newspaper articles, Allrecipes will remain in Seattle and the team will stay together, which was an important goal for many of us. And it will continue to provide a great food and recipe experience in the years to come, now enhanced by its association with Reader’s Digest and Rachael Ray.

Congratulations to my original co-founders and to the entire Allrecipes staff on an outstanding accomplishment!

Time off has been good, but of course it’s always better when you get to plan when and how to be off work, and not have events simply thrust it upon you. I have used the time to get plenty of exercise (it’s been a fairly nice fall), work in the kitchen quite a bit, and read a great deal. The latter isn’t yet reflected here in postings, but I’d finished book #48 when we shut down Network Clarity, and I’m just finishing up book #62 this weekend. This means I have yet to post on books #57-62, but I have a whole slew of drafts pending in Ecto and some notes will appear shortly.

Things have been quiet in November and December on Progressive Commons — we apologize for that but everyone has been busy with work and personal issues. I’m sure our writing will pick up again fairly soon.