Trial By Fire

Moving over to nextstopcafe.com

This blog has been stagnant for a while, but things are changing.  I've moved over to my own self-hosted blog that you can find at nextstopcafe.com, and I'll be blogging more regularly over there and will eventually retire this one.

Typepad was nice to get started, and it has loads of great tools if you are trying to become a "professional blogger".   But, I am not.  I blog for personal reasons, and hope sometimes people can find something useful in what I say, but I have no blogging ambitions and I will never use many of the tools Typepad offers.   More importantly, I'm a control freak and like to be able to run the software I want with the features I want.  I already have lots of server space of my own, and it was trivial to set up Wordpress and get it rolling over there on a platform I control and manage myself.

I might even finish that multi-part Microsoft article!  After three years, I believe it is more relevant than ever now that so many Linux-based products are nipping at Microsoft's heels.

So, stay tuned at nextstopcafe.com, I appreciate every single click.  Thank you.

Second Life Cynics Club

I vote for Clay Shirky as the president of the new Second Life Cynics Club because of his reactionary A Story Too Good To Check, just posted on Vallywag.  I agree there is a need for scrutiny, but Clay's article is just another paddle whacking at balls on the ping-pong table of hype.  Even so, his article is worth a read.  I agree totally that the hype about Second Life is ridiculous.  I also agree that the stories about Second Life visitors and "success stories" are greatly exaggerated.  But Clay's article shows how hype and confusion can affect clear vision of the negatives as well as the positives.

Unfortunately, I am not "good enough" to add comments to Valleywag myself (read their comment FAQ for proof that elitism is truly alive and well in the blogger world).  So, please check out Clay's article, and I'll add some of my comments here...

First, one thing Clay fails to clearly recognize is that virtual experiences are improving.  His justification for slamming Second Life involves a variety of comparisons to earlier incarnations of virtual environments.  In his reaction to Howard Rheingold's description of MUD's in 1996, he says:

This was a sentiment I believed and publicly echoed at the time. Per Howard, "MUDs are living laboratories for studying the first-level impacts of virtual communities." Except, of course, they weren't. If, in 1993, you'd studied mailing lists, or usenet, or irc, you'd have a better grasp of online community today than if you'd spent a lot of time in LambdaMOO or Cyberion City.

This reliance on hindsight ignores the possibility that people have indeed learned alot from failures of the past, and predicting the value of current virtual worlds based upon 12 year old outcomes is simply illogical.

His final paragraph highlights the main reason why I decided to comment:

There's nothing wrong with a service that appeals to tens of thousands of people, but in a billion-person internet, that population is also a rounding error. If most of the people who try Second Life bail (and they do), we should adopt a considerably more skeptical attitude about proclamations that the oft-delayed Virtual Worlds revolution has now arrived.

Clay's conclusion that "most of the people who try Second Life bail" may be true.  His posting comprises a lengthy set of explainations for why this might be.  But the biggest justification is mostly missed:  The motiviations of many new Second Life users are currently not based upon their desire to engage in a virtual world but rather they log-in because of their interest in the Second Life phenomenon itself.  And this interest is fueled by the hype of course.

The problem is observer bias.  Because of the hype, everyone expects to get a quick glimpse of this new incredible phenomenon.  They are disappointed, mainly because their expectations are based upon misinformation.  So, while the real Second Life enthusiasts continue to explore and discover a truly advanced virtual world experience, they are being deluged by "newbies" who have neither the time nor the incliation to engage in the activities needed to understand and appreciate Second Life.

This is having a tremendously negative impact on Second Life itself.  Consider:

1. Over 300,000 new users are signing up every 4-6 weeks.  Most of these are probably searching for the "secret" the press is talking about, and are cluttering up the environment.
2. Companies such as General Motors and McDonalds are taking their commercial brands into Second Life with ill-defined purpose.  Their investments are causing a supply-and-demand problem which is skewing the economics of island and private land purchases.   The result is that those creative individuals who have built the Second Life world are becoming disillusioned, and corporate "developers" are building pointless (but well funded) experiences because of a corporate investment mandate rather than true creative insight.
3. Strange side effects (such as the Second Life Liberation Army) are causing inworlders to change their behavior drastically because of hype-driven investment and traffic.  The SLLA, for example, hires "hitmen" to attack commercial brands who are "polluting" the purity of inworld life.

Human nature is so confounding.  Unless people calm down a bit, Second Life may well become yet another Internet traffic accident with curious bystanders loitering on the sidewalk as it slowly bleeds to death, or worse, goes public.  That would be tragic, because although there is clearly no "virtual worlds revolution", Second Life is one of the most remarkable steps heading in that direction.

Are YOU a music or video pirate??

Australia launched a newly updated "Music Industry Piracy Investigations" site today and sent us all an email which claimed that one of the big features of the site was helping you to determine if YOU are a music pirate. This type of site is yet another "let's criminilize the public" website, just like the SPA.

Featured on the site is an "Am I a Pirate?" page which invites us to ask ourselves questions which should, hopefully, reveal whether we have been infected with this little-understood disease.  Plus, they encourage us to report anybody we spot!

I think their list is missing some items.  So, in the interest of the public good, I put forward my own...

Questions to Ask Yourself If You Think You Might Be A Pirate

1. When iTunes tells you to insert any CD into your computer and makes it playable on your iPod, are you tempted to believe what you are told and click "yes"?
2. If your phone comes with software that lets you take a CD you own and make it into a ringtone, do you try it?
3. When you buy music from iTunes, and discover it won't play on your PSP, are you tempted to convert it to a format that allows you to play it?
4. When a television program you love is released two years later in your country than it is in the rest of the world, are you tempted to go to Google and search for it?
5. When you go to a band website and there is a music download, are you tempted to click on it without hiring legal investigators first?
6. After losing digitally purchased music several times because of "software bugs", are you tempted to make back up copies?
7. When a video starts playing on a MySpace page you visit, do you continue to listen, or instantly close your browser?

While I am certainly an advocate of legal media use, it's important to realize that the industry is sending consumers very mixed messages.  Stakeholders like Apple, Microsoft, and Nokia have a lot to lose if consumers "avoid" their new media features, so they encourage consumers to copy, encode, create ringtones.  In addition, none of the promises of technology are panning out for consumers.  So, people buy something, then discover the things they bought don't work where they expect them to.  No wonder people seek alternatives, and no wonder the integrity of "industry promises" is in doubt.  Consumers are just behaving normally, following the path of least resistance, and glad to open their wallets if they get something useful for a reasonable price.

Most people are inherently honest.  Let's start giving them credit for that.  If there are problems to fix, they're problems with the industry, not the consumer.

The Lower Echelon of Spelling

I care about spelling.  That does not necessarily mean I am good at it.  But, when I am about to type an unfamiliar word, I pause momentarily and ask myself "is that the correct spelling?" and will consult a dictionary.  Lather.  Rinse.  Repeat.  That's how my spelling improves.  I did this the other day when I was about to write the word echelon, and in doing so have concluded that everyone should avoid this word at all cost. I believe it triggers the yet-to-be-discovered "bad spelling gene" that infects so many careless spellers.

Let's start with simplicity itself.  The Echelon Corporation.  You can find them at www.echelon.com, just where you'd expect.  Nowhere on their FAQ do they mention how to spell their company name, nor do they make any apologies or explanations for choosing such a name.  Why should they?  They spell the word correctly.  I applaud them.

As far as I can tell, it seems they do not have a wholly-owned subsidiary called "Echelon Telecom", though even the FCC, Reuters, and the Department of Justice seem to believe they might.  They (and many others) refer to "Echelon Telecom" in financial, business, and legal documents.   Is Echelon trying to hide something?  Oh wait!  No, those are just mistakes!  They are documents written by educated people who know how to spell echelon.  But really, they meant to say "Eschelon Telecom"!

Eschelon Telcom (of Utah) has dibs on www.eschelon.com.  Here is where things get interesting.  When I went to their website, I was thankful that, right at the top, they have a link titled "Why Eschelon?"  Thank God, I thought, they're going to explain!  But, alas, nowhere do they answer the vital question "Why not Echelon?".  The page simply talks about their business and ignores the pressing issue of the name itself!  Nowhere on the "Eschelon's History" page do they talk about how they chose the name.  They only mention that it was founded in 1996 by Chairman Cliff Williams.

In fairness, if you look back in history, today's word echelon has its origins in the Old French word eschelon.  However, no dictionary I can find lists eschelon (or even eschalon) as alternative or even "incorrect" or "archaic" spellings.  The word is echelon, folks.  My guess is Cliff can't spell, or his branding advisors can't.  I claim that they chose their name poorly.

In defense of my claim, I call your attention to Docket 01-2263-01 issued by the Public Service Commission of Utah which "replaces a previous Order issued on October 4, 2001 and corrects the spelling of the Applicant's name".  What was the incorrect spelling?  Eschalon!  So, when the Utah PSC misspelled Eschelon, they tried Eschalon.  I at least give the Justice Department credit for "misspelling correctly".

Eschalon Telecom is in Utah, too!  Coincidence?  I don't think so.  A quick search of "utah spelling errors" in Google reveals the truth:  Utah has trouble with spelling.  It goes way back I think.  Early copies of "Life in Utah" (Beadle) when archived, require special considerations by the archivist such as "I have retained the original spelling for most of the incorrectly spelled words".  A blogger notes that fans at his Provo event did not know how to spell "Kick Their Asses" on the signs they were holding up.  Worse, while driving home after the game he encountered the road sign "Icey roads ahead".  And, as if that weren't enough, when searching for freelance proofreaders in Utah, we get the unfortunate result "0 Utah Proof Reading Specialists Found" from www.freelancedesigners.com.

Such tales, both historical and apocryphal, speak volumes for the state that not only spends the least per pupil on public education and had the highest percentage of voters bring Bush into office, but also gave us the "cold fusion" debacle.  Suffice it to say that I believe they're aware of the problem.  Provo knows about its misspelled street signs, and even mentions in their home page general policy that "Spelling errors will reflect on the image of Provo City".  It's unclear whether they meant that as a positive or negative.

But, I digress.  There may be worse offenders.

For example, what about Upper Eschelon Music Production Services at www.uppereschelon.com?  Is that an intentional misspelling designed to make us think they're "kewl"?  Ya think?

Certainly anybody trying to use Google to determine how to spell this word better look pretty hard.  If you search for "upper eschelon", you'll find thousands of apparently literate people who don't know how to spell the word.  I note the complete absence of the FCC, Reuters, and the Department of Justice in these searches.  They were the ones who felt compelled to refer to the Echelon corporation.  Maybe it renews my faith in government and media.  At least they can spell.

A word of advice, though.  If you're choosing a company name, why not pick one that people won't have any trouble spelling?  You'll save filing fees to correct misspellings, and avoid having a dozen or two Google searches for your name taking you elsewhere.  I know that I've probably opened myself up to criticism here.  Go ahead, I welcome you to check my blog entries for spelling errors.  I'm sure I've made a few outstanding mistakes.

If you report one, I'll be thankful.  I'll correct it.  At least I won't have to change my domain name.

iTunes and Online Music Buying

iTunes is fantastic.  It's also a total rip-off.  Last night it all became obvious to me when I, for the first time, made a rather innocent, typical consumer goof.  I accidentally bought something I didn't want.

Now really, it's my mistake.  I wanted ELP's Brain Salad Surgery, which iTunes doesn't have.  Instead, they have something called "Then and Now".  After sampling a track or two they sounded like the original tracks.  After purchase, I discovered that they really are a collection of live performances that are (to me at least) far inferior to the original recorded performances.  So much so, that I will probably never listen to them.

iTunes refund policy?  All sales final.

Now, If I had bought this at my local record store, I could do the following:

1. I could take it back and my local record store would give me my money back, especially if I had my original receipt and the disc was in like-new condition.
2. I could hold onto it until somebody I know turned up and I said: "Hey, do you want this ELP album?"  They would smile and say "thanks!" and greater friendship may result.
3. I could give it as a gift.
4. I could take it to a used record store and trade it.

At iTunes, I paid just about as much as I would have for a real CD.  Maybe more even.  And...

1. I cannot return it.  In fact, Apple's customer service policy front-line with respect to refunds is practically "We do not offer customer service.  Fuck you."
2. I cannot give it to anybody else.  The DRM'd files play only on my computer, and unless I authorized somebody else to play ALL my music, I cannot allow anybody who is not a family member access to the ELP album I just bought.
3. I cannot give it as a gift in any way possible.
4. There is no way I can trade it for something.

So, the score?  My record store gets four points.  iTunes zero.

By the way, I also use eMusic.  Because eMusic uses a trust model and ships MP3's, I at least have the ability to give my purchases as a gift.  Of course, the emusic selection isn't as good.  But, as an online music vendor, they clearly provide more value, and at a better price.

Now really, I love the online music buying experience.  But, Apple is really selling us crap and charging us a premium.  Ten years from now, when everyone looks hard at their iTunes DRM files, it will be more obvious.  For now, Apple wins.  Customers lose.

Back from Hawaii

Though nobody online knew it, I was away with Grace, Phil Tripp and Lisa Treen in Hawaii scuba diving for three weeks and got back at the beginning of July.  If you've been waiting for part three of my "Microsoft Series" articles, the  next one will be along in a few days.  I expected to have them finshed before going, but they turned into more of a project than I expected.

Stay tuned, and thanks for reading.

The Video iPod Doth Rock

I might be eating my words soon.  In a post last October I dared to make a prediction that "video iPod will be relegated to niche, specialty content, and the intense shift seen in music will never occur".  I even really emphasized the "never".  I should never make predictions, that's what I should never do!

It's not that the iPod video is setting the world on fire.  In fact, my prediction that it will never take off in the way music has may hold up, who knows.  However, Apple has created a "total mobile experience" that is full of so much raw potential that I can hardly believe every content provider isn't clamoring to produce mobile content for the iPod!  There is so much money sitting on the table and nobody is grabbing at it!  Wake up!

If you want to find out what the future will be like, take your iPod video and subscribe to "ABC News Now".  Every day, a 15 minute excellent video newscast that will make it clear that television news (on your television that is) is headed for the dumpster.  I am so impressed that ABC has done a dedicated news podcast.  Now the problem is, ABC isn't my favorite news program. I want every single news program to do the same thing.  I want choice.  Lots of choice.

And damnit, I'd be happy to pay!  I bet lots of people would.  That's the potential.

This difference between "potential" and "reality" is obvious if you take a look at a Google search for "iPod video".  Hardly anything of merit.  Mostly, it's outdated product reviews and Apple propaganda.  Now, try Googling for "iPod video torrents".  Ah ha!  A treasure trove of people talking about the real content on Bittorrent, Guba and other places.  These are the people who are experiencing the dream rather than just dreaming.

I confess, one of them is me.  I crossed the line.  And now that I have, here are some things you should try, if you dare:

First, go to www.podtropolis.com.  Hope you don't hear Police Sirens.  Hope it's still there when you read this.  Now, browse the shows.  Figure out the BitTorrent thing.  It's worth it.

Try Robot Chicken.  10 minute episodes you can watch in bed before you go to sleep.  The Simpsons do pretty well on the small screen.  Just browse, download, drag to your iTunes video icon, and sync your iPod.  Magical.  No complex software, no conversions.  The potential for short video is astounding, but nobody is doing it right in the commercial world.

Here's what commercial content owners need to do:

1. Create pay subscription feeds in iTunes using existing brands like ComedyCentral.  Make sure people get 3-5 podcasts each week in several feeds tailored to what people want.  Charge $4.95 per month.
2. Don't charge per-episode!  If you do, you're missing out on loyalty opportunities.  I would NEVER buy a single ABC News Now.  But, I'd subscribe to a whole month of them, no problem.  Same goes for Robot Chicken.  I would have never bought a single episode.   But, if it's part of the feed, I'd pay a monthly subscription.
3. Forget about on-demand streaming video.  I've been using on-demand video for 10 years.  Watching videos in "actively find, click, watch, navigate" mode is a waste of brain power and most people won't do it.  Focus on the iPod and the seamless subscription and delivery model.

Hurry up.  My credit card is glowing, and there's nobody to pay!   I certainly don't want to pay Guba.  They're just delivering (mostly illegal) content.  I'm the type of person that will gladly buy 3 boxed DVD sets per month.  I'll gladly pay for it online too.  Just make it possible!  Please.

Windows TNG: Use Linux as the Next Microsoft Platform

[Microsoft's New Win-Win Strategy: Post 3 of 5]

My last posting, "Scrap The Windows Codebase" explained some of the reasons why the current Windows codebase might be better off in the trash bin.  My mention of "Linux" caused myriads of knee-jerk reactions from other bloggers (see the Channel 9 forums, for example).  But, this post will only talk a little bit about Linux, and a lot about what might be needed for the future, and why the operating system has become too central a focus for Microsoft.

What IS Microsoft's Business?

I'm not sure Microsoft ever wanted to be in the operating system business.  There is been a lot of revisionist Microsoft history floating around, and the original goals of Gates, Allen, and eventually Microsoft are sometimes lost in the rhetoric of their successful business strategies.

Today, Microsoft knows their operating system is the cement that glues their business strategy together. 

Microsoft executives describe how desktop applications "widen the 'moat' that protects the operating system business". At the same time, Joachim Kempin in his 1999 testimony said "We are not in the operating system business.  We are in the computing business."  While at first glance this seems like defensive rhetoric to distract Jackson's team from their OS focus, keep in mind that Kempin was with Microsoft since 1983, and his chief responsibility was developing their relationships with PC manufacturers and OEM distributors.  Kempin, perhaps more than anyone else, saw the operating system as a useful business tool, one which wielded legendary power over extending Microsoft's computing products to every desktop.

While there is no doubt that this strategy has yielded tremendous financial rewards for Microsoft, it is nonetheless as strategy based, not upon technologies or innovation, but upon tie-ins, bundling deals, and partnerships.  In this sense, it limits Microsoft because, unlike other companies, they do not need to create the best products, only the most viable ones suitable for their (so far) successful business strategies.

But, when Gates is at his best talking about Microsoft's strategies, there is not a mention of predatory business practices, and the "moat" that protects the operating system.  Gates, in a notable 2003 interview with Fortune, said t "One was our vision, which has not changed since the day the company started."  The vision was, according to the interview, "the idea that you could buy PCs from many different hardware companies, and yet they would all run the same software".

C'mon Bill!  Let's look back briefly.  Bill originally thought computer languages were the company's business.  He went to IBM for an Operating System.  Only when IBM couldn't come to terms with Digital Research for an OS license, did Gates see an opportunity, buy QDOS, and license it back to IBM.  This sounds a bit closer to what Kempin might describe as "the vision": bundling and aggressive licensing.

All in all, I still believe Gates wants to serve the needs of consumers.  One of the advantages Apple has over Microsoft is that their vision of ease-of-use and style is strong by comparison to the muddled vision of Microsoft.  This wavering statement of vision, and inability to reconcile what Microsoft says with what it does is one of the reasons that so many Microsoft-bashers conclude that Microsoft is simply greedy and predatory.  Without clear vision, all successful business might appear so.

At his best, Gates talks of the dream "of a PC on every desk and in every home".  I think he still believes that comptuers are good for people and that the business mission is to do whatever is necessary to enhance the computer experience.

What does all this have to do with operating systems and my Windows TNG idea?  Well, two things.

First, I believe Microsoft has become so distracted by the importance of their operating system as the glue for their predatory business practices and the "moat" that they have stopped innovating.  Worse, the size and complexity of the operating system itself has slowed down even progress on Microsoft's business strategies.  If ever Microsoft needs innovation, now is the time.

The second point Bill makes better than I can.  In the Fortune interview, when asked "What can Microsoft do for small business?".  His answer, in full, is

Making our software simpler will probably have more dramatic impact with small business than anywhere.

Anyone who is following the current Vista releases knows that Microsoft is not moving toward simpler, but toward a far more complex, multi-faceted operating system.  The OS has taken over the company, and it's taken over the consumer's view of the product line.  Windows TNG is one perspective on how Microsoft might change that.

A New Platform

If Microsoft is going to return to the goal of extending the power of the PC and creating greater consumer value, they need a new platform.  What might that platform look like?

For a moment, let's speculate about the feel of a new product line, and the technical underpinnings.  Consider the profile of the products and technologies I'm suggesting, and don't get hung up dwelling on a particular flaw or inconsistency, as there will be many  This is, essentially, a "White Paper".

It's first worth considering exactly what the criteria might be for a new Windows.

Goal 1: Unbundling

Windows may not be too big if you consider all that it does.  And Windows may not be as monolithic as it seems.  Internally, there are many layers and boundaries.  But, it is delivered as a monolithic product, and because of that, organizational dependencies have been allowed to remain.  High level application changes, UI changes, and kernel changes all end up on one huge Gannt chart.  Despite attempts to avoid them, true development dependences exist between layers that simply should be separate.

What if, instead of a 50 million line Vista, we had the following:

1. A Windows TNG Kernel OS which has a separate release cycle, is developer-configurable, and was used in both desktop, embedded, and special-purpose applications by Microsoft and third party developers.  It would have drastically simplified security, a lightweight process model, and could be built in custom configurations by developers (similar to the way Windows CE is delivered).  Small configurations may have only a 400K footprint.  Large ones would have no larger than a 1.5M memory footprint.  (1-3 million lines of code)
2. A Windows TNG UI (essentially "Avalon-in-a-box").  Again, a separate release cycle, and custom-configurable.  This is a developer's product.  Because it is separate from the kernel, competing UI models can be developed and delivered separately.  Much of the "compatibility" with older Windows applications lies here.  (3-5 million lines of code).
3. A set of UI applications bundled with installers which is what users see as "Windows".  It installs and runs very much like Windows does now.  It is completely separate from the other two components, has its own schedule and can even be purchased in its unbundled form.  This is especially appealing for corporates who may want the important security or functional benefits of a new kernel but do not want to retrain users until later.  (5-10 million lines of code).

The new product has a technology profile shown in the (highly speculative) drawing at right.  Click on it for a bigger version.  Notice that:

1. The OS is unbundled, configurable, and separately shipped across all versions.
2. The Microsoft UI is optional, but shipped with the consumer product as standard.
3. On embedded devices, there is a more compelling case for vendor-specific UIs.
4. Specialized embedded devices, such as wearable devices, benefit from a very lean kernel with no GUI overhead.

Most importantly, all variants use the same kernel.

What happened to the other 32-41 million lines of code?  Well, they may exist somewhere, for example:

1. A product that might be called "Windows Legacy".  It's a virtual machine that runs a version of "Vista Minus" (or even "XP Minus").  Maybe it comes free with Windows, and is even installed by default for a while.  It will run 100% of all old Windows applications.  Thought it provides compatibility, it also sends a strong signal to customers that there is a dividing line beyond which compatibility may not be guaranteed in the future, and it serves to define where that line is at.
2. Specialized Server applications such as IIS, remote management software, etc.

Most of the savings comes from the next goal...

Goal 2: Reduced Complexity

Windows has become far too complex, and needlessly so.  For example, the past 15 years have seen 4 different phases of graphic support:

1. The pre-NT GDI model present in products such as Windows 95 and Windows 98.
2. The user-mode GDI model present until Windows NT 3.51.
3. The kernel-mode GDI and GDI+ model of NT 4.0 and XP.
4. The new Avalon framework for Vista.

While all of these are improvements, the compatibility requirement means that the legacy of these four models will exist for a long time.  This makes the product tremendously complex, holds back progress, and adds to the number of details applications programmers must learn.

Similarly, the Windows security model is orders of magnitude more complicated than Unix-based systems.  While Unix-based systems clearly are overly simple and not a good model either, the Windows model provides a detailed object-level security model which is far too expensive and yields few real benefits.  Rather than being used by developers, most security settings on most objects created in the Win32 API are left unchanged.  On a more pragmatic level, the weak passwords of most Windows users, coupled with the tendency of most non-corporate users to run as Administrator have rendered any security scheme irrelevant.

Vista is about to add yet futher layers of complexity on top of the already existing layers.  Localization, security, object management, and graphics are all about to be reinvented in Vista.  And yet, the old way will still remain.

Reducing the complexity of features such as these is essential, especially if upward compatibility is important.  The worst thing you can do for future systems is saddle them with complicated features that are superceded and replaced constantly as versions are released.

Reduced complexity can also have performance benefits.  For example, the Windows process model has always been criticized as "heavy".  Creating a process takes at least 10 times longer than creating a thread and frustrations over a complex process model can hamper efficient use of resources.  A new process model where threads and processes have the same weight, and processes can be created cheaply and easily would create greater performance opportunities in server-based applications.  This becomes especially important in real-time and embedded applications where lightweight processes are almost essential to development.

What about Compatibility?

Compatibility is one of the most difficult constraints in moving forward.  While Microsoft wants to say with 100% certainty that "Your application will run", the continual requirement of compatibility hampers progress on newer, superior technologies.  In addition, compatibility makes Windows itself less flexble.  If compatibility issues extend deep into the kernel and user APIs, then trying to deploy the Windows kernel in tiny embedded products will be almost impossible.

Microsoft will have to draw a dividing line in the sand, as Apple has done.  Applications on one side of the line will run with few if any changes, and this should represent about 90% of the applications developed in the past 10 to 15 years.  This means that some kind of "compatibility library" will need to be developed.

Isn't Microsoft Already Doing This?  Why start over?

Well, yes, and no.  While many features in Vista are targeted at these problems (such as the "Server Core" version of Vista), there is no true unbundling, and Vista will remain a monolithic product.  It will take years, or even decades, to gradually pull the pieces of Vista apart, and while those attempts are made, developers will continue to add more.  Attempts by Microsoft to truly create a layered operating system out of XP with no layering violations have been difficult.

Make or buy?

In many ways, what I'm suggesting is obvious.  Microsoft knows they'll have to replace Windows.  That's why projects like Singularity exist.  And, the goals I've spelled out above (including unbundling) may already be on the drawing board.

In theory.

In practice, the sheer size of Windows, and the compatibility juggernaut, will make everything take longer.  If Microsoft wants to replace Windows by 2015, it will take until 2025.  If they want it to be "fully compatible", then even Singularity will be hampered by the very same issues outlined in my previous post and this one.

Can Microsoft, and their customers, wait until 2025 until repeated evolutionary steps solve all the problems I've mentioned?  Will Microsoft continue to have such dominance that people will wait?  Unless Microsoft acts more quickly, it is inevitable that some competitor, probably Apple, will finally be able to attract large numbers of Windows users with an offering that, to users, appears similar enough to Windows from a purely functional perspective.

Maybe it's time for Microsoft to do what they've done so often before:  Acquire technology which solves the problem.

Here comes the L-word

Everything I've said until now has tried to make a case that:

1. The liability of the Windows codebase, including Vista, will slow Microsoft's progress to the point where vulnerability to competitors becomes threatening to Microsoft within the next 5-10 years.
2. Microsoft is solving these problems, but not fast enough for their users or shareholders.
3. It's worth considering if there are potential technologies which can be aquired to solve the problem.

Linux has the potential to solve Microsoft's problems, but it's important to look at the potential of Linux rather than the current reality.  Consider that Microsoft Visual Basic was originally purchased by Gates as Tripod by Alan Cooper, and SQL Server was written by Sybase originally until Microsoft negotiated exclusive rights to all Windows code.  What those products are today only a slight resemblance to what they were on the day they were purchased.

So, rather than look at Linux the way it is today, imagine what it would be like if Microsoft were to adopt the Linux platform, participate in and fund development, and drive the direction of Linux forward to meet Microsoft's own needs.

Technically, Linux has the following strengths:

1. It is a successful platform in use today, which is benchmarked and compared side-by-side with Windows.  In server-based applications, it often comes out ahead of Windows in some performance and security benchmarks.  Rather than being an "idea", it is an actual contender and is being used side-by-side with Windows in many corporate production environments.
2. Its security model is remarkably simple and even could be called antiquated.  Yet, for some reason, it has held up very, very well and is considered at least as secure as Windows.  Because it is so simple, it will be easy to upgrade and replace.  But, because it is working adequately now, replacement can be done carefully and cautiously while focusing energy on more important issues such as desktop innovation.
3. It has an efficient lightweight process model that is a superset of the one provided by Windows (that is, Windows process model can be built on top of the Linux process model).
4. It has been competing vigorously with Windows, and there is already a large device driver base.  In fact, vendors of hardware consider Linux to be second only to Windows in their priority for releasing device drives, and many vendors already do.
5. It has an entrenched development model which is popular in universities and many businesses.  thus, it is not necessary to spend significant time or effort on development products, especially for low-level drivers and server applications.
6. It has a configurable kernel which can be used in everything from tiny embedded devices up to very large multiprocessor systems.  The kernel is small, modular, and extremely robust.
7. It is much newer than Windows, and has very little legacy code by comparison.
8. An enormous amount of effort has already been done to create Win32 compatibility layers.  WINE, Crossoffice, and Xen are three specific technologies designed to run MS applications under Linux.  Rather than criticize these as inadequate and lame (they are), consider what would happen if Microsoft were to take over development of one of these.  Progress would be rapid and the problem of compatibility would be cleanly isolated.
9. It has a shared library model which allows multiple concurrent and incompatible versions of software to co-exist simultaneously without the need for extensive additional technology investment or developer education.

Linux has several weaknesses to consider:

1. The X-Windows platform is interesting, but outdated.  While a client-server windowing system has clear advantages, its API is more arcane and complex than Win32.  Microsoft (and the market) would be served well if Microsoft were to build (or buy) a new, more modern and capable graphics application framework for Linux.
2. The Open Source model requires a substantial investment in legal work and planning.  While some of the technologies would clearly remain Open Source, Microsoft would want to engineer as many components to be proprietary as possible.  This may dictate packaging and delivery "mechanics" in some ways.  Since many companies are already combining Open Source with proprietary products in their deliveries (such as Redhat), I am confident Microsoft can negotate this minefield in an aggressive and innovative way that would impress us all.

But, the biggest benefits of adopting Linux aren't technical at all...

Linux: The Business Reasons

While the technical issues can be argued to death, the business reasons for adopting Linux give Microsoft significant advantage.

Keep in mind that Microsoft has always been excellent at the adoption and assimilation of technologies that are already in the marketplace.  I remember when the Web was something Microsoft said they weren't interested in!  (Yes, it's true).  Despite the Java legal debacle (which Microsoft could have avoided, I believe), their adoption of Java was highly successful and if managed properly could have avoided having to "reinvent" replacements such as C#.

So, here are what I believe are the most compelling reasons why Linux is a good "buy" choice:

1. Microsoft completely eliminates open source as a competitor.  By embracing Linux, almost all open source efforts suddenly lose most of their "shared mission" to compete with Microsoft.
2. Microsoft extends dominance of the Office applications onto every desktop.  Efforts like Open Office, Sun Office, etc. become truly jokeware.
3. Competitors like IBM, RedHat and Sun start to shake in their boots as they realise that multi-million dollar investments they've been making in Linux have now directly benefited their most feared competitor.
4. Microsoft takes firmer control over how the GPL is applied to products.  Keep in mind that many products you buy contain both open source as well as proprietary components.  Windows TNG would be no exception.  But, because it was bread-and-butter to MS, they would apply their significant legal capability more productively.
5. The Open Source "religion" would become diffused.  Most open sourcers would be horrified to think that MS is "taking over".  As more and more MS successes occurred in the Open Source arena, the "we love Linux and software should be free" crowd become more and more marginal.

The Bottom Line

Sure, it's a minefield.  But, Microsoft is on a slow-burn right now to creating less and less competitive products while others are creating more innovative products with shorter delivery times.  By unbundling, creating leaner development strategies, adopting some proven technologies, and dominating the open source space, Microsoft can reinvent the entire industry.

That leaves Microsoft more time to outdo the very competitor who is making the greatest advances toward Microsoft's market: Apple.

The next segment of this post will explore how Microsoft might refocus and use their time to create a truly next generation desktop by creating a proprietary application layer on top of Linux and OS/X.

Being Negative about Windows

Ed Kaim responded to yesterday's "Scrap the Windows Codebase" post with some good comments and it's worth a follow-up.  Ed says "I was surprised by the negativity of the tone overall and felt it was very much in the style of Michael Moore".  Well, maybe I deserve that.  It is hard to talk about scrapping one of the industry's most valuable codebases in positive terms.

Ed has a good point when he says:

All I care about is that the OS does what my customers and I expect it to do and that the apps we build don't break. If it takes Microsoft 10 years to ship each new OS, that is better for us because it means less budget gets spent on migration and more on core projects. However, if the rug gets pulled out from under billions of users by drastic changes for questionable improvements, we're all screwed.

I agree with him 100%.  Microsoft has a overwhelming responsibility to their customers and shareholders not to cause needless market and consumer upheaval.

But here's the point:  It's far worse to live in denial.  If you have a problem, you need to face it full-on, even if it's more severe than you want it to be.  When a business has 10000 employees it can no longer use, it's not easy to make the decision to have massive lay-offs.  But it's a mistake if the business ignores the problem.

Re-read my post  on Windows Vista: Past Its Due Date Already, where I talk about this kind of denial in a similar situation with the product of a former software industry market leader:

Then a line is crossed.  You know that something is wrong.  Your engineers can feel it.  There's a malaise in the air.  But, nobody says anything.  At the lunch table, you read PC Week's scathing criticism. People stare around the room, some even laugh or scoff. Most say nothing.   You go back to your work, you immerse yourself in further enhancements to your product.  You convince yourself everything is OK. You look at competitive products only for purposes of punching holes in their strategy.  You find the holes.  You reassure yourself.  Everyone smiles.

Repeat until fail.

I recognize the pattern.  That's where Windows is right now.

A few MS employees have told me I'm not far off.  And Robert Scoble, in his short comment to the post, says "I totally agree".  Robert may not be on the team, but he's at least a close observer.

Ed also makes another good point:

There are smatterings of anti-Windows sentiment in broadly sweeping statements and quotations taken somewhat out of context that would indicate that people are fleeing Windows due to the problems Gary outlines. I don't see it at all.

He's right.  I don't think people are fleeing.  Windows customers want windows to be healthy.  Sure they do.   I was listening to an InfoWeek Podcast yesterday and Mitch Wagner said that the newest Vista Beta and Office 2007 have him ready to "eat his words" about former negative comments.  It's looking better, and we're all happy. 

Yes, even I am happy.  Nobody who relies upon Windows wants it to fail.  I'm not a Windows basher, trust me.  I did try Linux as my primary OS for 2 years.  I gave it a good try and ran my Windows apps under Wine or VMWare.  I should blog about it someday, it was an interesting experience in compromise.  When I switched back to XP 18 months ago, I felt like an old friend had returned.

When I was working on the dBASE project I talked about in my earlier post, it was the same way.  Everybody in the market wanted dBASE to be great.  Everybody inside Ashton-Tate felt that.  They wanted to produce the best product for the market.  Nobody was "leaving for other products".  There were no other alternatives!  Very much like Windows.  How can anyone leave?  There are truly no alternatives.

My upcoming posts will be less negative.  The last post was the "gosh we have a problem, Houston" post.  Of course it feels bad to admit the codebase is doomed.  Microsoft must eventually admit it.  But, my next post won't be "pro-Linux".  It will be pro-Microsoft.

Thanks for the comments Ed.  They'll keep me on track.

 

Windows TNG: Scrap The Windows Codebase

[Microsoft's New Win-Win Strategy: Post 2 of 5]

As a successful software architect, I've learned to recognize the results of a poorly managed design process.  In Windows Vista: Past Its Due Date Already, I gave some insight into why Windows matches a "software implosion pattern" I've seen before.   This post explores why Microsoft really should scrap the codebase, the next post will suggest the controversial idea that Microsoft should scrap Windows in favor of Linux.

The recognition that the code base is in trouble is no secret, and for the most part even Microsoft agrees.  Microsoft has a team called "The Windows Code Excellence Team".  They have a program for "driving broad changes efficiently into the Windows codebase".  They call these "Strike Force Efforts" and the very name they chose reveals the adversarial relationship that even Microsoft insiders perceive with their own code base.

But still, Microsoft thinks they can fix it.  This post is about why they can't.

The Recipe Was Wrong From The Start

The biggest single reason why Microsoft can't fix the problem is because Windows archtecture is flawed at its very foundation.  They're sinking money into repairs on an old car, trying to make it fuel efficient, trying to make it conform to current needs.  But as we all know, sometimes you need to buy a new one.

Understanding why this is true is technically challenging for most people.  The real issues are understood only by well-educated computer scientists.  As a result, most of the common discussions around Windows architecture talk about the visible aspects of Windows, rather than the technical underpinnings.  And, because the technical world is so full of geeks that argue pointlessly (a la SlashDot), there are rarely any efforts to bring clarity to Windows most severe problem.

There are ample sources of technical information about this.  While it may seem like a Windows vs. Mac argument, Daniel Eran's excellent (if lengthy) article "Five Architectural Flaws in Windows Solved in Mac OS X" is one of the best examples.  What can easily be missed about this article is that 4 out of 5 points Dan makes are directly the result of Apple's decision to use Unix as an architectural basis for OS X.  If Dan were talking about MacOS 9, only the first of his five arguments would apply.

So, by adopting Unix, Apple was able to push their own technology into the next generation.

Both anecdotal evidence and qualified research are available to illustrate the problem.  Security researchers can point squarely at flawed API design as a primary reason why Windows architecture actually "encourages insecure applications".  Those who remember Fredrick Brooks classic book "The Mythical Man-Month" see the same trip-ups in Vista that Brooks warned about in his book.

Because Windows architecture has fundamental design flaws, Microsoft is constantly adding layer after layer of new technologies to achieve what would be implicit in a better architecture.  This increases the amount of work, technology, and conceptual baggage that developers need to learn and master.  Thus, it decreases the relability of overall applications.  Developers have a limited amount of mental energy.  Any Windows developer knows that negotiating Windows "idiosyncrasies" takes up almost as much time as the application itself.

The idiosyncrasies themselves become entire "worlds of new technology"  Knowing how the Windows event loop operates, and arcane topics such as using "assemblies" as a way to avoid "DLL Hell" become distinct new art forms.  The Windows community has become so accustomed to this continual "band aid" approach that they don't even recognize the original problem.  Instead, the new technologies become job skills, and further serve to separate Windows programming expertise and culture from the larger world of shared computer science knowledge.  (See an earlier post of mine for more on this "disconnect").

Now and then, some Windows developer asks the right questions (from the InfoWorld gripeline):

I am surprised that still, after all these years, that Windows has not seen the solution that UNIX (and probably many OS's) takes to DLL Hell -- use versioned DLL files so something linked against an old DLL will use the old one while something linked against the new one will use the new one. Viola. Problem solved.

Continual layers of technology to solve architectural problems leads to the next problem with the Windows codebase ...

Escalating Complexity

In a recent New York Times article, the following appeared:

Several thousand engineers have labored to build and test Windows Vista, a sprawling, complex software construction project with 50 million lines of code, or more than 40 percent larger than Windows XP.

Windows is growing beyond even Microsoft's ability to manage it.    In October 2004, Martin Taylor, then Microsoft GM of Platform Strategy, admitted that changes were needed and introduced a new "role-based" strategy for reducing the size of the code-base:

"Today, it's still the entire code base. There's no reduction in the bits you get; things are just roped off," Taylor said Friday. "We want to get to a model of role-based deployment where you might just have the bits you need for that function. ... It's one of our design goals for Longhorn."

In that ComputerWorld article, Taylor was talking about a new agenda for Longhorn to trim the size of the code base.  The article later says

A Microsoft spokeswoman confirmed that the goals of providing a smaller Windows "footprint" are to cut maintenance costs and provide a "reduced surface attack area."

Martin is now part of the Windows Live team.  The new GM of platform strategy, Bill Hilf, hasn't said a word about it in the last 18 months.  So much for trying.  If a smaller code-base was one of the design goals for Longhorn, it seems Microsoft has decided to put the idea (and Martin Taylor) on the back burner for now.

Windows 3.1 had 2.5 million lines of code, Windows 95 had 15 million, XP has 40 million, and Vista will have at least 50 million.  Microsoft managers are stumped about how to reduce the complexity and size, and Michael Cherry, former Microsoft product manager, says "It's such a collection of smart people that they've started to believe too much in themselves". (MercuryNews)

Not only is the problem growing, but the team is looking the other way.

Reinventing Microsoft Software Culture

It's not enough to throw out the codebase.  Here's the real challenge:  You need to retool the very culture which allowed it to happen.  Before starting over, you need to figure out what (mis-)management style allowed Windows XP's excellent security foundation to be completely disabled by other arms of the organization.  Microsoft already tried to re-invent Windows once with Windows NT.  The problem is, while one set of OS experts in Microsoft is devising an excellent security framework, another set of "experts" is violating all the rules in the interest of "dumbing down the features" for users.  Security guru Steve Gibson (quoted in WindowsITPro) explains the phenomenon:

"With Windows 2000 you could argue that Microsoft was at least preserving the original NT security model," Gibson noted. "Regular users would log on as Administrator only when doing system tasks like installing applications or bug fixes, and then log on as a regular user to get work done. This is much like a UNIX machine, where the root account is tweaked very carefully, not generally used for day-to-day work. But Microsoft moved that NT security model to the home and gave Administrator power to users. [The company] discarded the traditional security model because it was too hard to explain to users..."

Aaron Margosis (a Microsoft employee) has an entire blog dedicated to the topic of trying to help people run Windows as "Non-Admin".  Despite his excellent advice, it is impossible for the average user to comprehend, much less follow such instructions when all of the default settings of Windows, and the expectations of third-party software are that average users will be running as Administrator.  As a result, Microsoft's excellent security model lies in the background, gathering dust, while clever hackers throughout the world know that it's open season for attacking the average user's desktop.  All of this is no accident, and is engineered into the product---introduced by well-meaning product managers attempting to make things simpler while elsewhere in the organization people know better.

It is as if Microsoft is releasing products by "trial and error".  They even recognize the "non-admin" problem and are moving to add yet another layer of complexity called User Account Control to Vista to help solve this problem.  But, even on the second attempt, it's not looking good.   Beta users are annoyed and claiming it is far too complex and intrusive:

In its current incarnation, too many people are likely to dismiss [User Account Control] completely, and if that happens, everyone loses. [Ed Bott -ZDNet]

That Microsoft has launched a major product with an major introduced security flaw is the most brazen sort of incompetence.  That they are still not getting it right reveals something much worse: ignorance.  While scrapping the codebase is essential, it's equally essential to establish new rules when moving forward.

Margolis final paragraph in one of his articles is among the most interesting.  Probably the biggest reason why Windows XP is so vulnerable, and that so many people run as root, is cultural:

Hey, y’all!  We need to lead by example.  People look to us for best practices, for the right way to do things. We are trying to convince the world that we are thought leaders in software and in software security. In the Unix world, they never run as root except when necessary. They “su”, do what they need to do, and revert back. We are not leaders when we run as root all the time. Comrades: you need to run as “User”, and your customers need to see you doing it. If you run into issues, don’t add yourself back to the admins group – file a bug against the offending product. Customers: if you see any MS sales, MCS, Premier, PSS, etc., doing web or email as admin, please tell them, “You’re not setting a very good example. I am disappointed.”

Spaghetti

While some experts agree that the above flaws are proven facts, I suspect there are more people in the industry that consider these things quite subjective.  As we all know, a picture is worth a thousand words.  In April, a ZDNet article with the dubious title "Why Windows is less secure than Linux" included two diagrams generated by Sana Security and shown below. These diagrams received little attention, but compare graphically how Windows and Linux process the service of a single web page.

The first diagram illustrates how Windows processes the page:

The second illustrates how Linux processes the page:

The orderly arrangement of the Linux traces are no accident.  They are the result of years of thinking which goes back to the origins of Unix itself.  Good operating system design assures that the operating system and application layers are distinct---separated by boundaries which are like immovable walls.  Such walls manage the compexity of systems by isolating operations from one another to minimize dependencies.  To a good system designer, these are not just guidelines, they are dogma.

To any good systems architect, the traces of the Windows diagram are like a giant black spot on your MRI.  They represent an undisciplined and haphazard set of interelationships resulting from years of unsystematic development and support of legacy code and processes.

Little Hope of Repair

There is hardly any hope of repair for such systems.  The new Windows Vista may eventually work, but it will be by brute force testing and a bit of sleight of hand---not good design.  Microsoft, however, is trying to fix it.  On his blog, Microsoft employee Larry Osterman describes the problem:

As systems get older, and as features get added, systems grow more complex.  The operating system (or database, or whatever) that started out as a 100,000 line of code paragon of elegant design slowly turns into fifty million lines of code that have a distinct resemblance to a really big plate of spaghetti.

This isn't something specific to Windows, or Microsoft, it's a fundamental principal of software engineering.  The only way to avoid it is extreme diligence - you have to be 100% committed to ensuring that your architecture remains pure forever.

It's no secret that regardless of how architecturally pure the Windows codebase was originally, over time, lots of spaghetti-like issues have crept into the  product over time.

Larry's right about the problem.  But unfortunately there is no way to turn back the hands of time and retroactively make sure that the architecture was pure from the start.  Yet, he goes on to describe how Microsoft has developed internal tools "that perform static analysis of the windows binaries and they work out the architectural and engineering dependencies between various system components".  The hope is that by knowing which layers should be isolated and why, changes can be put in place which fix the problem and eliminate the spagetti.

Then, unfortunately, Larry goes into the tall grass when he says:

Well, most of the layering issues can be resolved via email, but for some set of issues, email just doesn't work - you need to get in front of people with a whiteboard so you can draw pretty pictures and explain what's going on.

Software architecture may be an interesting thing to talk about in email or on the whiteboard.  But such naive attempts will not make the sweeping architectural changes that are necessary to yield noticable improvement.  Only good design, enforced by software tools and disciplined coding practices, can result in well-layered systems with managable complexity.  Much of the windows code itself predates the very tools and practices needed to fix it.  For the fundamental design of Windows to change, you need to go back to the drawing board.

Even if you believed, for a moment, that you could check every line of code and fix every dependency, the math would get you.  As the number of function points increases, the number of side-effects and dependencies increases exponentially.  Even a small software system can have millions of interdependent relationships.  A large system like Vista with 50 million lines of code would have side-effects and causal relationships that would defy analysis.

Confidence Building

In March, when Vista slipped, article after article appeared about the whys and whens of the slip.  The popular jourlalism moved quickly into an editorial stance.  The New York Times article "Burden of the years weighs on Windows" set the stage:

"Windows is now so big and onerous because of the size of its code base, the size of its ecosystem and its insistence on compatibility with the legacy hardware and software, that it just slows everything down," noted David Yoffie, a professor at Harvard Business School. "That's why a company like Apple has such an easier time of innovation."

Microsoft was uncharacteristically silent.  I believe, finally, there could be no disagreement.

Whether the problem is as egregious as I say, there certainly is the belief that it has reached a turning point in its lifecycle.  Consumer confidence in Windows behavior has waned, and now the recognition that the underlying operating system is to blame is becoming widely accepted.

If, as part of a bold new strategy, Microsoft announced that the Vista codebase was the end-of-the-line, confidence in future solutions could finally increase.   Instead of fighting the past, the talented teams of Microsoft engineers would be learning from their mistakes.  As it is, there is far too much code and far too many problems for them to do anything other than trudge forward, making it work as best they can.

Conclusions

The Windows codebase is in bad shape.  It's unlikely that Microsoft, or indeed anyone, can fix it.  I am certain they will create a usable version of Vista.  But, I expect that one year after its release, we will not be looking back, happy that the problems are solved.  Instead, such an albatross of design can only yield new problems, and new challenges for Microsoft.

Get rid of it, and replace it.  But with what?  In the next post (coming in a couple days), I'll suggest that Linux be a part of a new strategy to revitalize the product line.  It's good for Microsoft in more ways than you think.