William Buckley on General Franco

"General Franco is an authentic national hero. It is generally conceded that he above others had the combination of talents, the perseverance, and the sense of righteousness of his cause, that were required to wrest Spain from the hands of the visionaries, ideologues, Marxists and nihlistis that were imposing on her, in the thirties, a regime so grotesque as to do violence to the Spanish soul, to deny, even Spain's historical identity."

http://www.slate.com/id/2185301/pagenum/2

While we're at it, another quote, this time from Richard Nixon: "General Franco was a loyal friend and ally of the United States."

http://en.wikipedia.org/wiki/Francisco_Franco

When I am saying that in the US we're having a choice between fascists on the right, and the center-right party on the "left", people are thinking that I am exaggerating. I am not.

Article on mobile platforms...

...that I might have written, but this guy beat me to it :-).

http://blog.7touchgroup.com/2010/01/windows-mobile-iphone-android-marketplace-comparison/

Another pay vs. performance article

http://www.huffingtonpost.com/2009/12/31/high-ceo-pay-may-correlat_n_406578.html

Your faith is not strong enough...

http://www.ebonmusings.org/atheism/guestessays/religion101.html?

Sr Oracle needed at Bank of America

...presumably so that they could predict the bursting of the next bubble ahead of time.

30000 new troops for Afghanistan - yeeeeeeehaw!

Web applications!

2 browsers (IE and Firefox) running Ajax.org demo are maxing out T9300 on my laptop - a Core 2 Duo running at 2.5 GHz. This is plugged in.

Try it yourself:

http://www.ajax.org/public/presentation/tae/presentation2.html#home

Best pie chart ever - directly from Fox (via Reddit)

Modern capitalism

The top 5 execs at Lehman Brothers made $1B from 2000 to 2008. The top 5 execs at Bearn Stearns made $1.4B during the same period.

"The whole idea of capitalism is that the people provide the capital and the executives take care of it for us. In this case, the people provided the capital, and the executives took it."

http://crooksandliars.com/susie-madrak/lehman-bros-bear-stearns-ceos-walked

Victim of car accident is tragically not Glenn Beck

Victim In Fatal Car Accident Tragically Not Glenn Beck

Meet Britain's Dick Cheney!

The photo in this article is priceless...

http://www.guardian.co.uk/business/2009/oct/21/executive-pay-bonuses-goldmansachs

Matt Taibbi on naked short selling, or what killed Bear Stearns

Really awesome article on naked shorts. There are only a handful of news organizations who do real journalism these days, and, surprisingly, Rolling Stones seems to be one of these few. I am also very impressed with Bloomberg reporting recently - it looks like after Murdoch bought WSJ the best people went to Bloomberg.

http://www.rollingstone.com/politics/story/30481512/wall_streets_naked_swindle/print

Here's the really scary thing - we've lost the integrity of our financial markets.

The reason US was so successful in attracting capital was the transparency with which financial markets here used to operate, and strong regulations that ensured this transparency - for the most part, the legacy of Great Depression. However, over the last couple of decades these regulations have been weakened, and now we're back to the turn of the century stock swindles and off the books accounting (http://www.creditloan.com/blog/mark-to-market-accounting-changes-favor-banks/).

All this - coupled with the dollar fall - can mean only one thing: US will certainly lose its status as a financial capital of the world within the next decade.

In other news, marijuana legislation has been relaxed (http://finance.yahoo.com/news/AP-Newsbreak-New-medical-apf-4109207182.html?x=0&sec=topStories&pos=main&asset=&ccode=). So instead of worrying about markets, botched wars, and disappearing industrial base, the population can just get stoned! Finally, a government that knows how to rule...

Government rationing of health care

There were all these inane reports in the press about "Obama's suicide panels" and government rationing the health care - all widely covered.

Let's see what they say about this beauty - an insurance company says that they will only cover a woman if she gets sterilized - because she had a C-section in the past.

Under the hill

Check out where you can deliver a birthday present on 1-800-flowers.com!

Think pay for performance works? Think again...

http://www.ted.com/talks/dan_pink_on_motivation.html

This TED talk alludes to a number of social experiments where adding monetary incentive has significantly decreased the productivity of creative work.

Extremely enlightening, a must watch for every manager - and yet another proof that the claim that the Wall Streeters need to be paid exorbitant amounts of money to retain "the best and the brightest" is just a crapload of bull.

United States v. 8,800 Pounds, More or Less of Powdered White Egg Product

"United States v. 8,800 Pounds, More or Less of Powdered White Egg Product, et al., No. 07-3671, 2008 U.S. App. LEXIS 26098 (8th Cir. Dec. 24, 2008)(defendant’s importation of egg whites from Peru without first obtaining a pasteurization certificate from the Peruvian government and without obtaining authorization from the Food Safety Inspection Service violated the Egg Products Inspection Act, which could result in the condemnation and destruction of the egg whites; trial court’s granting of summary judgment upheld)."

It's hidden somewhere around the middle of the list on this page: http://www.calt.iastate.edu/aglawrl.html

I kid you not!

Big Pharma and research

I love it when they say that the pharmaceuticals are so expensive in the US because the companies need this money to finance research. In reality, if you look at a typical big pharma income statement, you will discover that only ~20% of its expenses are research - the rest is mostly sales and marketing.

But here's another proof beyond any reasonable doubt that what big pharma is all about is sales - its donations to political parties. Researchers giving money to repubs? Basically, the same people who say that Earth is 10,000 years old, and dinosaurs walked the planet alongside people, radiocarbon dating be damned?

This political contributions profile is more in line with what you expect from used car salespeople than the research and development organizations...

http://www.goodguide.com/contributions#sector=Pharmaceuticals



By comparison, here's how the software sector looks like:

http://www.goodguide.com/contributions#sector=Computer/Internet&sort=total&query=

Bravo, Apple! You have class!

Apple quits the U.S. Chamber of Commerce over its ‘frustrating’ global warming denialism.

http://thinkprogress.org/2009/10/05/apple-quits-chamber/

TFS 2010 will be much easier to install

I've spent a week trying to get TFS 2008 installed on my home infrastructure. It was the most painful piece of software that I've ever installed. And it looks like I was not the only one who felt that:

"Installing TFS has been a pain point for years. Although it’s gotten better, 2010 represents a quantum leap. The TFS installer now has 3 wizards: Basic, Standard and Advanced. The big innovation is the new “Basic” install wizard. It is a Next, Next, Next install experience that allows you to install and configure TFS in about 20 minutes or less (assuming .NET and SQL Express are already on your computer – a little longer if TFS has to install them for you). Both will already be there if you’ve installed VS 2010. The Basic wizard will install and configure IIS (if it’s not already there), install and configure SQL Express (if it’s not already there), and install and configure TFS. The only thing that really pains me is installing .NET 4.0 requires a reboot :(."

http://blogs.msdn.com/bharry/archive/2009/10/01/tfs-2010-for-sourcesafe-users.aspx

John Olivier on Afghanistan

Afghanistan as the last level of a videogame "Empire"!


Absolutely priceless!

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
The Unwinnable War in Afghanistan
www.thedailyshow.com
Daily Show Full Episodes Political Humor Healthcare Protests

Guns and morons

I am slowly building up my collection of World War II and former communist block firearms. This involves scouring the internet with searches on certain keywords.

Recently, I had a sad realization - most of the US gun-owning community is not mentally qualified to operate a fork, rather than own a gun. Which may explain an extremely high rate of gun-related death in the United States - 30896 in 2006.

Here's a representative post (http://answers.yahoo.com/question/index?qid=20080920100316AAlVQMH):

"Should i buy an SKS, AK-47 or AR-15 rifle?

the cheaper the better, but not if it is total garbage. The purpose of owning one for me is to have a assault rifle before they are illegal. i want something that can pack a punch when the world goes to crap so i can defend my family. yes, the purpose would be to kill, so go somewhere else hippies, this is a hypothetical situation i would like to be prepared for if it ever happens."

"Tactical advantage", "firefight" are the terms that litter the gun boards, so instead of answering these questions in dozens of places, I decided to come up with one meta-response here.

"Dear Moron,

Let me try to answer your question from the liberal/hippie point of view, since this angle is typically not covered on the gun boards you are frequenting.

First of all, this is the stupidest thing I read on the internet today! Why, you might ask? Well, there are multiple reasons.

First, do you really expect that hippies will attack you and your family? Hippies? Seriously? Like this guy?


Second, if us hippies/liberals wanted to attack you, surely we wouldn't storm your house? Did you know that there is a strong positive correlation between education and liberal views? You might not realize that, but one skill that they teach in college is thinking. So if we really did want to get you, we would surely be able to devise a better approach.

For example, we could just wait outside your house, and spray you with bullets from a safe distance when you come out to buy groceries. Or if we wanted to force the events, why not setting your house on fire and then just shoot everything that comes out?

But in reality, we wouldn't even bother with this at all. We'd just send a black helicopter (http://en.wikipedia.org/wiki/Black_helicopter). As you know, being government/UN freaks, we have plenty at our command. It would take one missile to have your house look like this, from a safe distance:


Do you really think that AK-47 vs AR-15 would make a material difference here?

Finally, even if you really did get into one of the "firefights", I bet you would not have much luck in a "tactical combat situation". I've played with your kind in Halo. You will be the biggest target, in the center of the field, collecting all the bullets.


I bet like with any other profession, it takes years of hard work to train a soldier - not an act of buying a gun.

So take my advice - instead of wasting money on something that you aren't mentally qualified to operate and won't be able to use, buy a book. Or sign up for a history or biology class at your local community college. You really could use the extra IQ so that next time people won't look at your writing and say - Geez, this is the stupidest thing I've seen on the internet today!"

Beowulf

The Queen Anne Blockbuster is going out of business, and, among other things, I picked up a copy of Beowulf (http://www.beowulfmovie.com/) for a couple of dollars.

Boy was this a weird experience! So weird that, in a sense, it deserves to be a cult movie - a la Striptease. For most of the movie we couldn't tell if these guys were being serious, or trying to spoof something. I am still not entirely sure. We laughed throughout big part of it.

From a really overdone dialog ((think of Gimili’s boasting in The Lord of the Rings, multiply that by 800, and you have the most humble of Beowulf’s pronouncements), to nude Angelina Jolie's feet, that seamlessly merged lack of shoes with high heels, to the way they were hiding - in a really conspicuous way - Beowulf's genitalia as he was fighting Grendel in the nude (this was very, very similar to the scene where Bart rides through the streets of Springfield in Simpsons The Movie) - it looked more like a comedy than what the movie was trying to present.

As a comedy it might have been mildly funny, though not hysterical. If you enjoyed Striptease, watch this one. Otherwise, it’s probably not worth the hour and a half.

If you don't like George Bush, you should take a look at his voters

http://www.telegraph.co.uk/news/worldnews/northamerica/usa/6173399/Charles-Darwin-film-too-controversial-for-religious-America.html

"The film was chosen to open the Toronto Film Festival and has its British premiere on Sunday. It has been sold in almost every territory around the world, from Australia to Scandinavia.

However, US distributors have resolutely passed on a film which will prove hugely divisive in a country where, according to a Gallup poll conducted in February, only 39 per cent of Americans believe in the theory of evolution."

An oldie...

...but a goodie!

"Cheney Waits Until Last Minute Again To Buy Sept. 11 Gifts"

http://www.theonion.com/content/news/cheney_waits_until_last_minute

How much for shipping, again?

Check out the Fedex Next Day rate on this $10 battery...


Think this is egregiously expensive? It is, but it still comes out way ahead of our local Best Buy where a similar (actually, less powerful) battery retails for $39.99 plus tax...

Classy Sarah Palin fans react to Kennedy's death

Just look at the spelling in these gems. The correlation between the level of education and political views is very real...

"thank you for maintaining my belief in you as a real american, however this country is now much better off, one less socialist, anti freedom senator."

"Now if we could just talk God into taking Arlin Spector, Harry Reid,and Nancy Pelosi America would be Eutopia!"

"good riddens"

"If he makes it into Heaven (& I doubt he will with his stance on abortion) I hope that God makes him babysit all the aborted children for eternity. God have mercy on his soul."

"Ted Kennedy dying has made my day...."

"He cannot fillibuster God. Good ridencance to a sorry person."

"It's about time, we can only hope Pelosi and Ried will be joining him very soon. All 3 of them should be buried in Moscow for whom they work so tirelessly."

http://www.alternet.org/blogs/peek/142220/sarah_palin%27s_facebook_%27friends%27_celebrate_ted_kennedy%27s_death%3A_%22one_less_socialist%2C%22_%22good_riddens%22/

"Brilliant" marketing

I love it when people say that Microsoft has terrible products which only succeed because of our amazing marketing.

http://www.pcworld.com/article/170820/microsoft_apologizes_for_racially_charged_image_alteration.html

I had no idea so many people hate Java...

...so passionately!

http://www.reddit.com/r/programming/comments/9dzpu/ask_reddit_why_does_everyone_hate_java/

All Hail His Noodly Appendage!

Introducing Black Square, a MS Word document review system

Black Square is to specs what Malevich is to code. This release is 0.1 - treat it as a preview, it has not been field-tested at all. There are known important bugs. But if you do feel adventurous, give it a whirl!

http://blacksquare.codeplex.com/

Big thanks to Eric White (http://blogs.msdn.com/ericwhite/) who coded the most important part of the system - the document merger.

For more background on Black Square, see the blog post where it was originally introduced: http://1-800-magic.blogspot.com/2009/07/lockingunlocking-word-doc-files.html

Physics lessons in Alabama

Driving to work today I was listening to yesterday's Marketplace podcast on my Zune.

They were reading a piece about schools in Alabama having big trouble finding teachers in math and sciences. Imagine recruiting a biology teacher who'd have to teach "alternatives" to evolution! - must be super hard indeed.

So they are importing teachers from Philippines. One school mentioned in the show had its first physics class in several years(!) and it had just 17 people enroll. No wonder such a staggering percentage of US population is convinced that Earth is 6000 years old...

Just say no to political correctness

'Something strange has happened in America in the nine months since Barack Obama was elected. It has best been summarised by the comedian Bill Maher: "The Democrats have moved to the right, and the Republicans have moved to a mental hospital."'

http://www.independent.co.uk/opinion/commentators/johann-hari/johann-hari-republicans-religion-and-the-triumph-of-unreason-1773994.html

We liberals brought this upon ourselves. We should stop acting like the idiots' point of view is entitled to respect and equal treatment in the media. It's not. The cooks, the crooks, and the cons - including the neocons - need to be called for what they are, loud and clear.

In particular, we should stop trying to prove that science and religion are somehow compatible. They are not.

Science requires preponderance of evidence. For science, if something is unobservable in principle, it does not exist.

Religion requires faith in tales written thousands of years ago by semi-literate shepherds, claiming the miracles that have never been observed in practice.

Most Americans are big boys and girls - it's OK to have them face the hard choice.

Either you believe in science and enjoy the fruits of labor of these godless scientists (http://people-press.org/report/?pageid=1550) - including antibiotics, computers, jet travel, and sanitation. Or choose supernatural and go back to the middle ages and 40 years average life span.

Trying to have it both ways is like trying to lose weight without a diet. It simply does not work.

Dude, have you heard of Netflix?

I am channeling the Fake Steve Jobs now, but damn it, two insane press stories a day...

> That's largely because they're not a darn thing worth watching or
> playing that uses Moonlight/Silverlight. Go ahead visit the
> Silverlight site; let me know when you find something compelling. I didn't.

http://blogs.computerworld.com/14570/moonlight_2_arrives_and_falls_flat_on_its_face

Netflix of course uses Silverlight for streaming the part of its content that's available online.

A pundit tries OpenGoo. Hilarity ensues.

OpenGoo is an open source counterpart to Google Docs. The idea is that you download and host it on your own servers. This C|Net reporter gets really, really confused by this concept:
http://news.cnet.com/8301-13505_3-10310817-16.html

In case they remove this article, here's the screen shot (click on it to read):


This is even funnier than Investor Business Daily's "Steven Hawking would have no chance under British health care" piece... I am wondering, what kind of college degrees do these people have? And what kinds of colleges graduate these "journalists"?

British healthcare

Apparently there's been a bunch of ads on TV feeding US populace the horror stories about how horrible NHS is.

This is what real British - you know, the ones actually living there and using British health care (where doctors by and large are government employees and the health care costs are almost entirely paid by the government) - have to say about it:

"Watching these debates is like reading National Geographic. It's just impossible, from a European perspective, to understand what these people are on about. Their political views seem as backwards and removed from the world we live in as a shaman casting magic spells."

http://www.politics.co.uk/analysis/health/comment-i-ll-never-understand-americans-$1318652.htm

Also, from Investor's Business Daily editorial on 8/3: "People such as scientist Stephen Hawking wouldn't have a chance in the U.K. where the National Health Service would say the quality of life of this brilliant man, because of his physical handicaps, is essentially worthless."

After much ridiculing on the interwebs, they removed the sentence. Pity, it was yet another proof that money != brain.

Couple of interesting links for Win7 setup

Booting from VHD:
http://blogs.msdn.com/knom/archive/2009/04/07/windows-7-vhd-boot-setup-guideline.aspx
Referenced in there, but worth a separate mention, using USB stick for Win7 setup:
http://www.elmajdal.net/Win7/Installing_Windows_7_From_a_USB_Stick.aspx

Bingin' Malevich

I've done a bit of egosurfing today (http://en.wikipedia.org/wiki/Egosurfing) and found that my code review system (http:/malevich.codeplex.com) shows up as #2 when searching for Malevich in Bing.


Frankly, I was surprised.

By comparison, Google search has it towards the bottom of the second page. Mondrian - Google's own code review tool which inspired Malevich - is at the very bottom of the first page.

Now, there was a fresh batch of conspiracy theories fomented by the "technical" pundits that claim that Microsoft uses Bing put down Apple (http://www.pcworld.com/article/169750/bing_search_reveals_promicrosoft_results.html). To me this is like a toddler pushing a nuclear submarine to help it go faster: Windows certainly does not need help from Bing to compete with Apple.

I think there is a very simple explanation for both Malevich and Mac vs PC phenomena. The search engine certainly uses clickthrough data to guide its searches. Bing's biggest market share in the geek population is Microsoft employees - there are tens of thousands people here that use Bing. The overall market share of Bing is small enough so that a few thousand users can swing the search results considerably - bringing Malevich-the-code-review-system (which is also extremely popular here) and articles critical of Apple on top.

Admittedly, this is not nearly as juicy as the "evil Microsoft adjust the search results to favor itself" conspiracy theories, but it appears to be the simplest one :-).

http://en.wikipedia.org/wiki/Occam's_Razor

GI Joe, the movie

No, I have not seen it. But I've read Roger Ebert review of it, which is a masterpiece on its own.

http://rogerebert.suntimes.com/apps/pbcs.dll/article?AID=/20090807/REVIEWS/908079997

"The two teams also each have a skilled Ninja fighter from Japan. Why is this, you might ask? Because Japan is a huge market for CGI animation and videogames, that's why. It also has a sequence set in the Egyptian desert, although there are no shots of dead robots or topless pyramids. And Cobra headquarters are buried within the miles-deep ice of Arctic. You think construction costs are high here. At one point the ice cap is exploded real good so it will sink and crush the G. I. Joe's submarine. We thought ice floated in water but, no, you can see big falling ice chunks real good here. It must be only in your Coke that it floats."

Developer Connection

If you thought (like I did) that TFS documentation was bad, check out this beauty:

http://developer.apple.com/documentation/appleapplications/Reference/WebKitDOMRef/DOMSelection_idl/Classes/DOMSelection/index.html

As a mental exercise, try to guess what does empty() do? Does it empty the selection? Or does it return true if the selection is empty?

Obama's birth certificate - a Kenyan angle!

FROM: Mr. James Thambo,
Email:jamesthambo@workmail.co.za
jamesthambo@msn.com
TO: Orly Taitz, Esq (Orly.Taitz@aol.com)

KEEP AS CONFIDENTIAL

Dear Mrs. Orly Taitz:

I am Mr. James Thambo, a Barrister to US President Barrak Hussein Obama's great great uncle Matimor Thambo Hussein. He has died 3 days ago after being sick with Cancer, and now I am in Charge for Executing his In-heritance. Before he died he gave me a Birth Certificate for US President Barrak Hussein Obama issued By Kenian Republic in 1961 that Proves that Barrak Hussein Obama is not a US Citizen. It is Numbered 47O44 and Executed with All Appropriate Authority. I will give you the name of my bank where said Certificate is stored and other important information if I receive a positive reply from you.

I want you to be my partner, to secretly transfer the Certificate of Birth to the United States where it would be Sold on ebay. My business Partners here has estimated the Auction Value for the US President Barrak Hussein Obama's Birth Certificate to be $100,000,000 (U. S. Dollars!). I cannot Sell it myself Because I am a Barrister to the late US President Barrak Hussein Obama's great great uncle Matimor Thambo Hussein and they would suspect me if I sell it myself. All you would have to do is place this Item for bid on Ebay, and receive the money into your account.

Your share will be 30% which is $30,000,000 (U. S. Dollars!). My own share will be 69%, which is $69,000,000. We shall keep 1% which is $1,000,000 for expenses. Reach me immediately by mail so that I can give you further details. Also provide me your direct tel/fax to reach you, and your bank account number, and your credit card so I can Certify the Authenticity.

Thank you and God Bless,

Mr. James Thambo.

------------------------

http://www.huffingtonpost.com/2009/08/03/kenyan-birth-certificate_n_249850.html

Joys of contextual advertising

Removing duplicate comments from a word document

As I wrote before, I am working on Malevich-like system (http://malevich.codeplex.com) for reviewing specs in the same way we're reviewing code.

This work is based on Eric White's excellent blog post about merging comments from two identical files (http://blogs.msdn.com/ericwhite/archive/2009/07/28/merging-comments-from-multiple-open-xml-documents-into-a-single-document.aspx).

The idea is to have a web site where one uploads a Word document, the reviewers then download a locked copy of it which only allows adding comments. They then use Word to comment, and upload the files back. The server merges all comments (using Eric's code) back into the master copy. Every person who downloads the document afterwards gets the comments from all previous reviewers.

While working on this system, I had to add two things in terms of comment management.

First, I had to lock files so only adding comments is allowed. The code for this is here: http://1-800-magic.blogspot.com/2009/07/lockingunlocking-word-doc-files.html.

Second, Eric's code merges the comments by adding all comments from one document to the other. Unfortunately what this means is that after the very first reviewer has added his or her comments, every time someone else downloads the copy with these comments, adds more, and uploads the document back, the original set of comments gets duplicated. So I had to write code that cleans up this duplication.

The comments in the Word files leave in a special section accessible through MainDocumentPart.WordprocessingCommentsPart.Comments of the WordprocessingDocument class. They can be enumerated as follows:

WordprocessingDocument doc = WordprocessingDocument.Open(args[0], true);
foreach (Comment c in doc.MainDocumentPart.WordprocessingCommentsPart.Comments)
    Console.WriteLine("{0} {1}:{2}", c.Id, c.Author, c.InnerText);

This section contains the comments themselves, but it does not have any information as to where the comments attach to the actual text in the Word document. Instead the comments attach via commentRangeStart, commentRanveEnd, and commentReference elements that are intersperced into the text of the paragraph:

<w:p>
  <w:r>
    <w:t xml:space="preserve">This is a test</w:t> 
  </w:r>
  <w:commentRangeStart w:id="0" /> 
  <w:commentRangeStart w:id="2" /> 
  <w:commentRangeStart w:id="4" /> 
  <w:r>
    <w:t>document</w:t> 
  </w:r>
  <w:r>
    <w:rPr>
      <w:rStyle w:val="CommentReference" /> 
    </w:rPr>
    <w:commentReference w:id="0" /> 
  </w:r>
  <w:commentRangeEnd w:id="0" /> 
  <w:r>
    <w:rPr>
      <w:rStyle w:val="CommentReference" /> 
    </w:rPr>
    <w:commentReference w:id="2" /> 
  </w:r>
  <w:commentRangeEnd w:id="2" /> 
  <w:r>
    <w:rPr>
      <w:rStyle w:val="CommentReference" /> 
    </w:rPr>
    <w:commentReference w:id="4" /> 
  </w:r>
  <w:commentRangeEnd w:id="4" /> 
  <w:r>
    <w:t>.</w:t> 
  </w:r>
</w:p>

To the developer, these elements are accessible from the root element of the Word document's MainDocumentPart:

foreach (CommentReference cRef in
        doc.MainDocumentPart.RootElement.Descendants<CommentReference>())
    Console.WriteLine("Found reference for {0}", cRef.Id);

foreach (CommentRangeStart baseRs in
        doc.MainDocumentPart.RootElement.Descendants<CommentRangeStart>())
    Console.WriteLine("Found range start for {0}", baseRs.Id);

foreach (CommentRangeEnd baseRe in
        doc.MainDocumentPart.RootElement.Descendants<CommentRangeEnd>())
    Console.WriteLine("Found range end for {0}", baseRe.Id);

Unlike the beauty of almost Lisp-like functional code that Eric wrote to merge comments, the code below goes through some contortions trying to determine that comments that have the same text and author really do start and end in the same place of the Word document. Location is important in determining the equivalence of comments because it is easy to imagine a whole bunch of separate, different comments with the same text, for example, "Here, too.", that would otherwise be considered equal.

To compile the code, you need to get and install Microsoft's OpenXML SDK 2.0 from here: http://www.microsoft.com/downloads/details.aspx?FamilyId=C6E744E5-36E9-45F5-8D8C-331DF206E0D0&displaylang=en, and add a reference to DocumentFormat.OpenXml assembly which the SDK installer puts in GAC.

Here's the code. It is rather self-explanatory: it collects all the relative elements from the document - comments, ranges, and comment reference points, determines which ones are duplicates, then removes the dupes.

There is subtlety that this code relies upon which appears to be true, but technically does not technically have to be - that for the comments that are attached to the same location the commentRangeStart and commentRangeEnd elements have the same sequence - e.g. if comment A's commentRangeStart preceedes comment B's commentRangeStart, then comment A's commentRangeEnd should preceed comment B's commentRangeEnd. While this seems to be true for Word, if you are adopting this code for general purpose OpenXML, I would recomment changing the logic to remove this dependency.

//-----------------------------------------------------------------------
// <copyright>
// Copyright (C) Sergey Solyanik.
//
// This file is subject to the terms and conditions of the Microsoft Public License (MS-PL).
// See http://www.microsoft.com/opensource/licenses.mspx#Ms-PL for more details.
// </copyright>
//----------------------------------------------------------------------- 
using System;
using System.Collections.Generic;
using System.Xml.Linq;

using DocumentFormat.OpenXml;
using DocumentFormat.OpenXml.Packaging;
using DocumentFormat.OpenXml.Wordprocessing;

namespace RemoveDuplicateComments
{
    /// <summary>
    /// Removes duplicate comments in an OpenXML document.
    /// </summary>
    class Program
    {
        /// <summary>
        /// Removes duplicate comment in an OpenXML document.
        /// </summary>
        /// <param name="args"> Command line arguments (file name). </param>
        static void Main(string[] args)
        {
            if (args.Length != 1)
            {
                Console.WriteLine("Usage: removeduplicatecomments filename");
                return;
            }

            Dictionary<int, Comment> comments =
                new Dictionary<int, Comment>();
            Dictionary<int, string> commentTexts =
                new Dictionary<int, string>();
            Dictionary<int, CommentRangeStart> commentRangeStarts =
                new Dictionary<int, CommentRangeStart>();
            Dictionary<int, CommentRangeEnd> commentRangeEnds =
                new Dictionary<int, CommentRangeEnd>();
            Dictionary<int, OpenXmlElement> commentReferenceParents =
                new Dictionary<int, OpenXmlElement>();
            HashSet<OpenXmlElement> commentReferenceParentsSet =
                new HashSet<OpenXmlElement>();
            HashSet<int> idsOfIdenticalStarts = new HashSet<int>();
            HashSet<int> idsOfIdenticalEnds = new HashSet<int>();

            WordprocessingDocument doc = WordprocessingDocument.Open(args[0], true);
            foreach (Comment c in
                doc.MainDocumentPart.WordprocessingCommentsPart.Comments)
            {
                Console.WriteLine("{0} {1}:{2}", c.Id, c.Author, c.InnerText);
                int id = int.Parse(c.Id);
                comments.Add(id, c);
                commentTexts.Add(id, c.Author + " : " + c.InnerText);
            }

            foreach (CommentReference cRef in
                doc.MainDocumentPart.RootElement.Descendants<CommentReference>())
            {
                Console.WriteLine("Found reference for {0}", cRef.Id);
                commentReferenceParents.Add(int.Parse(cRef.Id), cRef.Parent);
                commentReferenceParentsSet.Add(cRef.Parent);
            }

            foreach (CommentRangeStart baseRs in
                doc.MainDocumentPart.RootElement.Descendants<CommentRangeStart>())
            {
                Console.WriteLine("Found range start for {0}", baseRs.Id);

                int baseId = int.Parse(baseRs.Id);

                commentRangeStarts[baseId] = baseRs;

                string baseCommentText = commentTexts[baseId];

                CommentRangeStart rs = baseRs;
                for (; ; )
                {
                    CommentRangeStart next = rs.NextSibling() as CommentRangeStart;
                    if (next == null)
                        break;

                    rs = next;

                    int rsId = int.Parse(rs.Id);
                    if (baseCommentText == commentTexts[rsId])
                        idsOfIdenticalStarts.Add(rsId);
                }
            }

            foreach (CommentRangeEnd baseRe in
                doc.MainDocumentPart.RootElement.Descendants<CommentRangeEnd>())
            {
                Console.WriteLine("Found range end for {0}", baseRe.Id);

                int baseId = int.Parse(baseRe.Id);

                commentRangeEnds[baseId] = baseRe;

                string baseCommentText = commentTexts[baseId];

                CommentRangeEnd re = baseRe;
                for (; ; )
                {
                    OpenXmlElement nextEl = re.NextSibling();
                    while (nextEl != null && commentReferenceParentsSet.Contains(nextEl))
                        nextEl = nextEl.NextSibling();

                    re = nextEl as CommentRangeEnd;
                    if (re == null)
                        break;

                    int reId = int.Parse(re.Id);
                    if (baseCommentText == commentTexts[reId])
                        idsOfIdenticalEnds.Add(reId);
                }
            }

            foreach (int id in idsOfIdenticalStarts)
            {
                if (idsOfIdenticalEnds.Contains(id))
                {
                    Console.WriteLine("Eliminating comment {0}", id);
                    commentRangeStarts[id].Remove();
                    commentRangeEnds[id].Remove();
                    commentReferenceParents[id].Remove();
                    comments[id].Remove();
                }
            }

            doc.MainDocumentPart.RootElement.Save();
            doc.MainDocumentPart.WordprocessingCommentsPart.RootElement.Save();

            doc.Close();

            Console.WriteLine("All done!");
        }
    }
}

Apple is replacing Microsoft as a company Linux advocates love to hate

Of course, there's still plenty of hate for everyone... still, so much fun to watch!

http://www.defectivebydesign.org/blog/jailbreaking-apple-iphone

Apple is really, really, really not a developer-friendly company

http://www.riverturn.com/blog/?p=455

Locking/unlocking Word doc files programmatically

My team is going through a planning milestone again, and this means reading, reviewing, and approving a lot of specs and design documents.

So for this weekend I was toying with the idea of setting up a clone of Malevich (http://malevich.codeplex.com) for document reviews.

Malevich is of course the tool we (and now a whole bunch of other teams inside and outside Microsoft) are using for code reviews. Its main target is to make commenting easy - you simply click on a line of source code, an edit box opens, you type your comment for that line, and that's it. You can read more about Malevich's inspirations and aspirations here: http://1-800-magic.blogspot.com/2009/01/malevich-introduction.html.

Over the last 7 months Malevich has proven to be a big success. It streamlined code review process in the development team, involved many people in code reviews who otherwise would not be participating, and did wonders for the quality of our code base.

All this made me start thinking about introducing a similar process for spec reviews. After all, a review is a review, right?

The biggest problem with the spec reviews turns out to be the file format. Malevich operates on text files, and so rendering these files on the screen, showing a difference between the two versions of a file, and associating comments with the line turns out to be very simple. Specs (at Microsoft) are traditionally written as Microsoft Word documents.

Word turns out to have a very nice commenting mechanism, but rendering documents on a web page is not nearly as straightforward, and diffing them... that's a whole another project!

While pondering this idea, I ran into this blog post by Eric White: http://blogs.msdn.com/ericwhite/archive/2009/07/05/comparing-two-open-xml-documents-using-the-zip-extension-method.aspx which describes how to determine if two Word documents are the same (modulo comments). The post served as my first introduction into OpenXML, which is the format behind the Word document. Also, I read that Eric was planning a blog post about merging comments from two documents, and this lead me to the following design for the spec review site.

I am going to put together a system very similar to Malevich (let's call it Black Square for now), but instead of text files, it would hold Word documents. To create a review request, a reviewee would upload a document to the server via a web site. Upon upload, the server will lock the Word file in a way that would prevent all modifications to it other than the comments. It will then make the document available for reviewers to download.

To perform a review, the reviewer downloads the document, comments on it using Office reviews functionality, and upload it back to the server. The server will then merge the comments back into the master document, making comments from everybody available to all subsequent reviewers as well as the reviewee.

I've shot Eric an email, and as it turned out, he had already largely completed his merger, and he gave me a preliminary copy to beta test (the final version is now here: http://blogs.msdn.com/ericwhite/archive/2009/07/28/merging-comments-from-multiple-open-xml-documents-into-a-single-document.aspx).

Then I spent part of the weekend coding. After a few hours I had a skeleton web site and needed to code the first meaningful action - locking a Word document so only comments could be added.

When I have to deal with large new API sets, I tend to program by Google - search for a code snippet that best illustrates the use of the API. Internet is a great resource for that (with the only exception - reading is fine, copying code with unclear copyright into commercial problems is not!), and Windows source is even better (although I cannot use that for the open source projects, for similar reasons).

Well, as it turned out, there is a dearth of samples when it comes to OpenXML programming. Unlike most of .NET APIs, MSDN has no examples of use in its API documentation. There are a few "How to" samples of solving and end-to-end problem which primarily focus on processing the text, not configuration options of the Word file. And the rest of the Internet is pretty much silent on the subject.

To make matters worse, the API is based on XML with a bunch of types derived from base XML elements, so Intellisense does not often works.

After some struggle (and help from Eric) I was able to make sense of the programming model. Here's what's going on here.

The document has a bunch of sections. You can look them up by changing the docx extension of the file into zip, and then opening it in your favorite archiver. You will find that the file is just a zipped archive of a bunch of XML files. What I've done to figure out what elements need to be changed to lock the file was making the copy of the file, expanding it, then locking the file, expanding the result, and then diffing it.

This led me to two elements: documentSecurity in properties of ExtendedFilePropertiesPart, and documentProtection. The first one was easy - it had a counterpart in the object model, "doc.ExtendedFilePropertiesPart.Properties.DocumentSecurity", setting it was very easy:

WordprocessingDocument doc = WordprocessingDocument.Open(args[1], true);
doc.ExtendedFilePropertiesPart.Properties.DocumentSecurity =
new DocumentFormat.OpenXml.ExtendedProperties.DocumentSecurity(isLock ? "8" : "0");
doc.ExtendedFilePropertiesPart.Properties.Save();
doc.Close();

The second was a setting in MainDocumentPart. The hiccup for me (a very novice XML developer - remember, most of my life was spent deep in the guts of OS, I have not touched managed code and all attendant goo until a few months ago!) was that settings were a collection of OpenXML elements, and DocumentProtection, despite the existence of the type, was not addressable in the direct way, as a property of the settings. Instead, the settings needed to be interpreted as an XML record, e.g. via LINQ to XML:

DocumentProtection dp =
    doc.MainDocumentPart.DocumentSettingsPart.Settings
    .ChildElements.First<DocumentProtection>();
if (dp != null)
    dp.Remove();

if (isLock)
{
    dp = new DocumentProtection();
    dp.Edit = DocumentProtectionValues.Comments;
    dp.Enforcement = DocumentFormat.OpenXml.Wordprocessing.BooleanValues.One;

    doc.MainDocumentPart.DocumentSettingsPart.Settings.AppendChild(dp);
}

doc.MainDocumentPart.DocumentSettingsPart.Settings.Save();

So here's a full code snippet. It gives you a command line utility to lock and unlock Word files (unlocking the file will - I think - also remove the password protection, although I did not try this).

You need OpenXML Format SDK 2.0 to run this, available here: http://www.microsoft.com/downloads/details.aspx?FamilyId=C6E744E5-36E9-45F5-8D8C-331DF206E0D0&displaylang=en, and a reference to DocumentFormat.OpenXml in your project.

//-----------------------------------------------------------------------
// <copyright>
// Copyright (C) Sergey Solyanik.
//
// This file is subject to the terms and conditions of the Microsoft Public License (MS-PL).
// See http://www.microsoft.com/opensource/licenses.mspx#Ms-PL for more details.
// </copyright>
//----------------------------------------------------------------------- 
using System;
using System.Xml.Linq;

using DocumentFormat.OpenXml.Packaging;
using DocumentFormat.OpenXml.Wordprocessing;

namespace LockDoc
{
    /// <summary>
    /// Manipulates modification permissions of an OpenXML document.
    /// </summary>
    class Program
    {
        /// <summary>
        /// Locks/Unlocks an OpenXML document.
        /// </summary>
        /// <param name="args"></param>
        static void Main(string[] args)
        {
            if (args.Length != 2)
            {
                Console.WriteLine("Usage: lockdoc lock|unlock filename.docx");
                return;
            }

            bool isLock = false;
            if (args[0].Equals("lock", StringComparison.OrdinalIgnoreCase))
            {
                isLock = true;
            }
            else if (!args[0].Equals("unlock", StringComparison.OrdinalIgnoreCase))
            {
                Console.Error.WriteLine("Wrong action!");
                return;
            }

            WordprocessingDocument doc = WordprocessingDocument.Open(args[1], true);
            doc.ExtendedFilePropertiesPart.Properties.DocumentSecurity =
                new DocumentFormat.OpenXml.ExtendedProperties.DocumentSecurity
                (isLock ? "8" : "0");
            doc.ExtendedFilePropertiesPart.Properties.Save();

            DocumentProtection dp =
                doc.MainDocumentPart.DocumentSettingsPart
                .Settings.ChildElements.First<DocumentProtection>();
            if (dp != null)
            {
                dp.Remove();
            }

            if (isLock)
            {
                dp = new DocumentProtection();
                dp.Edit = DocumentProtectionValues.Comments;
                dp.Enforcement = DocumentFormat.OpenXml.Wordprocessing.BooleanValues.One;

                doc.MainDocumentPart.DocumentSettingsPart.Settings.AppendChild(dp);
            }

            doc.MainDocumentPart.DocumentSettingsPart.Settings.Save();

            doc.Close();
        }
    }
}

BTW, for the not faint-of-heart, here's the documentation for OpenXML format: http://www.ecma-international.org/publications/standards/Ecma-376.htm

And here are the Microsoft SDK docs: http://msdn.microsoft.com/en-us/library/bb448854(office.14).aspx

How did Adam and Eve's kids have kids? (via Reddit)

How to display a Word document on a web site

Couple of interesting links:

(1) An anatomy of an OpenXML document is here: http://msdn.microsoft.com/en-us/magazine/cc163526.aspx

(2) OpenXML format SDK: http://www.microsoft.com/downloads/details.aspx?familyid=ad0b72fb-4a1d-4c52-bdb5-7dd7e816d046&displaylang=en

(3) Finally, an example of using said SDK: http://blog.maartenballiauw.be/post/2008/01/11/Preview-Word-files-%28docx%29-in-HTML-using-ASPNET-OpenXML-and-LINQ-to-XML.aspx

Windows 7 and Server 2008 R2 go RTM!

Build 7600 is it, and it goes on MSDN August 6!!!

Freedom and the Bible

"Romans 13:1-7 (NLT): Everyone must submit to governing authorities. For all authority comes from God, and those in positions of authority have been placed there by God. 2 So anyone who rebels against authority is rebelling against what God has instituted, and they will be punished. 3 For the authorities do not strike fear in people who are doing right, but in those who are doing wrong. Would you like to live without fear of the authorities? Do what is right, and they will honor you. 4 The authorities are God’s servants, sent for your good. But if you are doing wrong, of course you should be afraid, for they have the power to punish you. They are God’s servants, sent for the very purpose of punishing those who do what is wrong. 5 So you must submit to them, not only to avoid punishment, but also to keep a clear conscience. 6 Pay your taxes, too, for these same reasons. For government workers need to be paid. They are serving God in what they do. 7 Give to everyone what you owe them: Pay your taxes and government fees to those who collect them, and give respect and honor to those who are in authority."

About blowjobs (via Reddit)...

http://seminal.firedoglake.com/diary/6314

Among all the idiocy printed today about Chrome OS

...finally, the voice of reason! Ladies and Gentlemen, I give you... fake Steve Jobs!

http://fakesteve.blogspot.com/2009/07/lets-all-take-deep-breath-and-get-some.html

The mother of all bull...

"Google Drops A Nuclear Bomb On Microsoft. And It’s Made of Chrome."

http://www.techcrunch.com/2009/07/07/google-drops-a-nuclear-bomb-on-microsoft-and-its-made-of-chrome/

The idiots in the press are at it again, cooking a sensation by blowing up an interesting tidbit of information way out of proportion.

Let me point out two obvious facts.

(1) The entire consumer market is rather small as a share of Microsoft revenue (10%?). The netbooks most likely represent less than 1% of the company's revenue stream. You cannot possibly call a "nuclear bomb" something that targets so little money.

(2) The smart phone market will always be much bigger than a netbook market. So if the "nuclear bomb" metaphor made any sense, Apple has dropped it years ago with iPhone.

Here's another stupid quote of the day:

'"One of Google's major goals is to take Microsoft out, to systematically destroy their hold on the market," said Mr Enderle.

"Google wants to eliminate Microsoft and it's a unique battle. The strategy is good. The big question is, will it work?"'

http://news.bbc.co.uk/2/hi/technology/8139711.stm

When I was at Google, the last thing people there were thinking about was Microsoft. I maybe have heard Microsoft mentioned a grand total of 10 times in my year plus there. What Googlers do care about is building cool things that attract attention and make customers come to their sites. THAT strategy clearly works. Destroying Microsoft - not so much (Netscape tried that approach).

My own take on this - thank you, Google! Windows 8/IE 9 will be better for your efforts. It often takes a competitor to persuade us that a segment of a market is important (unfortunate, but true). With this announcement Google did just that.

Do you have a health insurance?

Don't be so sure. You might lose it when you actually need it. Apparently, insurance companies slap a $1M surcharge on corporate policies that carry expensive patients. The companies then face a choice of whether to essentially pay you a $1M+ salary or...

http://www.dailykos.com/storyonly/2009/7/7/751100/-How-I-lost-my-health-insurance-at-the-hairstylists

Incidentally, in 3/4 of all medical bankruptcies (which are half of all bankruptcies in the US) people had health insurance.

http://1-800-magic.blogspot.com/2009/05/us-healthcare-by-numbers.html

BMI is bogus... because it embarrasses USA

It was making sense up to a point where an author claimed that 200 years ago most people led sedentiary life styles, although I had to ignore his quip on "if the formula does not describe the data, rig the formula" (this, of course, is what science - at least theoretical physics - is all about).

But when I got to the end, it was this: BMI does not make sense because...

"10. It embarrasses the U.S.

It is embarrassing for one of the most scientifically, technologically and medicinally advanced nations in the world to base advice on how to prevent one of the leading causes of poor health and premature death (obesity) on a 200-year-old numerical hack developed by a mathematician who was not even an expert in what little was known about the human body back then."

http://www.npr.org/templates/story/story.php?storyId=106268439&sc=fb&cc=fp

Come to think about it, an even more ridiculous fact is that our entire space program is based on a 300-year-old formula developed by a theologian!



This pearl of logical reasoning comes to you directly from a Stanford (!) Professor (!) of Mathematics (!) Keith Devlin...

http://www.stanford.edu/~kdevlin/

P.S. The author of this blog takes no position on the validity of BMI as a measure of human obesity, only on the validity of the referenced above argument against it.

Scriptster: C# as a scripting language

I love Python! Unlike the vast majority of script languages that evolved ad-hoc, Python was built in a controlled way and has features, syntax, and runtime which click together.

I learned it at Google, got readability in it (http://1-800-magic.blogspot.com/2008/01/after-8-months-no-longer-noogler.html, and have written a few thousand lines of code since. I must say that of all the scripting languages, Python is probably the most amenable to writing thousand-plus line programs (maybe with the exception of Ruby).

As far as I am concerned, it would be nice if all other command interpreter scripting died (sorry, PowerShell) and Python were integrated into shells everywhere.

There is one problem with Python, however - it is yet another language to learn, and yet another development/runtime environment to maintain. Its integration with Windows is good, but not nearly as good as C# which was literally made for Windows. And so after I came back to Microsoft (http://1-800-magic.blogspot.com/2008/06/back-to-microsoft.html) and found myself doing most things in C# and C++, I started getting more and more rusty with Python.

Of course, being a dev manager does not help - my opportunities to code are few and far between.

What that meant that scripting became harder and harder. Soon I found myself writing small executables in C# instead of scripts. Which was all good, except that you end up with two things - a source file and an executable, which now need to be maintained together, checked in together, etc. And for very small things, it's a considerable overhead.

And then I had an idea. .NET, you see, ships with a C# compiler in the box - it is present on every (updated) Windows system. What if I were to write a small program, a C# "interpreter" of sorts, that would run C# programs directly from a command line as if they were batch files?

And so Scriptster was born.

Scriptster is a single executable that allows you to run C# programs directly, without manually compiling them. It automatically compiles C# code before running (and caches the compiled versions so the next execution is faster), but to the user it is completely transparent. It is fast, too - even the first, compiling, invocation takes fractions of a second. Subsequent runs are instantaneous.

Scripter is copy-deployable: simply copy it into a directory of your choice, run

    scriptster --install

(from a command prompt started with administrator privileges - "Run as Administrator"), and close and reopen CMD windows where you expect to be using it (CMD needs update to PATHEXT environment variable, which it reads on load).

After this, you can author and run C# scripts. For example, open Notepad, and type the following:

using System;

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("This program was invoked with the following command line parameters:");
        foreach(string s in args)
            Console.WriteLine("{0}", s);
    }
}

Save this file as testscript.csscript (the files need to have .csscript extension to work with Scriptster), and you can run it directly from the command line:

    c:\temp> testscript Blah blah blah
    This program was invoked with the following command line parameters:
    Blah
    blah
    blah

You can run considerably more involved scripts with Scriptster of course, in fact, any C# program can be run, as long as it is (1) in one file, and (2) only relies on assemblies in GAC.

Here's an example of a program that queries LDAP for user alias:

//#ref System.DirectoryServices.dll
using System;
using System.Collections.Generic;
using System.DirectoryServices;
using System.Text;

class Program
{
    static void Main(string[] args)
    {
        DirectorySearcher ds = new DirectorySearcher();

        ds.PropertiesToLoad.Add("mail");

        foreach (string alias in args)
        {
            ds.Filter = "(SAMAccountName=" + alias + ")";
            SearchResult result = ds.FindOne();
            if (result == null)
            {
                Console.Error.WriteLine("Could not resolve {0}", alias);
                continue;
            }
            Console.WriteLine("{0}'s email is {1}", alias,
                result.Properties["mail"][0].ToString());
        }
    }
}

Notice "//#ref" at the top of the file? This is how you tell Scriptster about referenced assembly.

You can edit scripts in Visual Studio, debug them, then rename the files to .csscript extensions and run them right from the command line. You can take existing small programs, and run them from the command line, too.

Interested? You can download Scriptster, including the source code, from CodePlex: http://scriptster.codeplex.com.

Give it a whirl, and let me know how it works out.

The story of Mel - oldie but goodie

http://www.pbm.com//~lindahl/mel.html

Jeff Immelt (GE's CEO) on the road ahead

http://www.businessweek.com/the_thread/economicsunbound/archives/2009/06/immelt_speech_u.html

Obama has the Ring...

• Indefinite detentions: check!
• Broad executive powers: check!
• Secrecy: check!
• Signing statements: check!
• Escalation of foreign wars: check!
• ...

http://sipseystreetirregulars.blogspot.com/2009/04/obama-and-ring-of-power.html

Couple of consumer guides

A must read for information on how to use your credit card (fine print translated into human-readable form)...

http://www.mint.com/blog/finance-core/the-descent-into-credit-card-debt/

... and on how to use your iPhone...

http://www.reddit.com/r/WTF/comments/8w094/att_charges_adam_savage_of_mythbusters_11k/c0alsiz

"A friend of mine used to work for AT&T customer service. He had a call one day from a guy working for a very small company. A company size of five people, in fact. All of them decided to get iPhones and get a shared plan. Then they all decided to go out of the country...apparently for over a month.

Well, they spent a little less than a month over seas when one of them called my friend to ask some innocuous support question. They had not yet seen their bill of over $300,000. My friend did not say a word to him, but hung up and laughed his ass off.

Originally I thought this story was unbelievable, and I doubted it. I found it was possible after some rough calculations, but still it was kind of an extreme case.

Now, I see more and more of these stories popping up. Five people on the same plan, out of country, and all using their iPhones extensively...I'm starting to believe."

ActiveDirectory and disk imaging: not a happy combination

When I first heard about Hyper-V snapshotting I was extremely excited. This feature allows one to freeze an image of the virtual machine's hard drive (take a snapshot), and revert back to it at any point in time.

Moreover, it supports a snapshot tree: you can install the OS, take a snapshot, install one application, take a snapshot, revert back to the original image, install another application, and, again, take a snapshot. As a result, you now have three images which you can boot at any point in time (although not simultaneously): clean OS, app1 install, and independent - and clean - app2 install.

If you ever had to test your software in multiple environments, this is an absolute Holy Grail.

So I did it and it worked - for a while.

Unfortunately, one of the security features of the NT domain is that machine accounts periodically (once a month) change their passwords. This is driven by the client, not AD server (as described here: http://blogs.technet.com/askds/archive/2009/02/13/machine-account-password-process.aspx - which is a good introduction on how machine passwords work), and can - in theory - be turned off. But it's on by default, and is probably on as a security policy at most actual corporate installations.

So in a month the currently running version of VM changes its password. Which then renders all the rest of the snapshots useless: they have the old passwords. If you boot any of the snapshot, your VM can no longer connect to the domain. If you disconnect and rejoin, it gets a new machine SID and a new password. Which means that the password (and SID) of the version of the VM that was running previously - and all other snapshots, as a matter of fact, - is now bad.

All this means that after one month, the snapshot tree that you've just invested so much time building becomes completely useless.

The problem is not limited to Hyper-V per se. It manifests in every imaging solution - Vista/Server 2008 backup, Norton Ghost, etc. The only way to fix it - if your domain policy allows it - is to disable password change. Which brings us back to the link I mentioned earlier.

http://blogs.technet.com/askds/archive/2009/02/13/machine-account-password-process.aspx

The difference between United States and Soviet Union...

...might be that in the US most people wholeheartedly believe the propaganda, where's in SU most people didn't. As far as the foreign policy goes, the actions of the two countries seem to be about the same.

http://digbysblog.blogspot.com/2007/04/truths-consequences-by-digby-since.html

There is a duality of 1984 and Brave New World. In 1984 the government rules because everybody is afraid. In Brave New World it rules because nobody cares. But the net result is still the same.

New US Motto: We are too big to fail

http://freakonomics.blogs.nytimes.com/2009/02/24/the-new-six-word-motto-for-the-us-is/

Open source in China

This place: http://ostatic.com/blog/actuate-survey-open-source-booming-in-china-germany-and-other-regions says that 80% of Chinese use Open Source software.

Yet this impromptu Google survey says that only 8% of people around Times Square know what a browser is: http://1-800-magic.blogspot.com/2009/06/oh-customers.html.

Doesn't compute, does it? People who don't know what a browser is would know about Open Source and be able to distinguish it from other forms of software licenses?

I think I know what's going on here: 80% of Chinese users think that Windows and Office are open source products because they got pirated copies of them for free :-)!

We're going to be proud of our OS again!

This is Win7 running on a VM with Visual Studio and an instance of SQL server. The CPU spikes are compiles. Notice 0% CPU and only 1.32GB of RAM.

Cost of crapware in battery life

I watched a presentation about power last week. Among other interesting numbers in it - a clean install of Vista has less than 1% of CPU utilization on idle; an image from an OEM (including a bunch of 3rd party software) had ~7%. An increment of 10% CPU utilization leads to 8% less battery life...

Oh, customers... Part Deux.

A picture is worth 1000 words...

But 1000 words is roughly 5K, and a moderately-sized picture is at least 50K. Plus - unlike pictures - the text is searchable. Go figure...

Talking to a wall... (via Reddit)

In Jerusalem, a female journalist heard about an old Jew who had been going to the Western Wall to pray, twice a day, everyday, for a long, long time. So she went to check it out. She goes to the Western Wall and there he is! She watches him pray and after about 45 minutes, when he turns to leave, she approaches him for an interview.
"I'm Rebecca Smith from CNN. Sir, how long have you been coming to the Western Wall and praying?"
"For about 50 years."
"50 years! That's amazing! What do you pray for?"
"I pray for peace between the Jews and the Arabs. I pray for all the hatred to stop and I pray for our children to grow up in safety and friendship."
"How do you feel after doing this for 50 years?"
"Like I'm talking to a fuckin' wall."

http://www.reddit.com/r/atheism/comments/8tkf8/in_jerusalem_a_female_journalist_heard_about_an/

Oh, customers...

I kept maybe five textbooks from college. One of them was Ordinary Differential Equations (via Reddit)

http://www.reddit.com/r/AskReddit/comments/8sios/who_was_the_best_teacher_youve_ever_had_what_made/c0aamlv

I hate reposting, but this is too good to leave it up to Reddit's comments retention policy. Reproduced for posterity. But if you like it, do go to the link above and upvote...

By kleinbl00:
"The guy on the left.

He was a graduate from the University of Zagreb or something and he had an awesome accent. And he was beanpole tall and twitched around. He was like a cross between Cosmo Kramer and The Count from Sesame Street.

He was incredibly passionate about what he tought. He would bang on the chalkboard, breaking the chalk, and say "DoyounderstandTHIS? DoyouGETthis?" and look at us all intense. And then he'd roll on with what he was going.

The dude fucking loved math. He was teaching ordinary diff EQ and you'd think he was Beethoven explaining crescendos. And he really didn't give a shit about homework. He'd sit there and drill us through stuff as if our life depended on us understanding. I saw that guy tear up a couple times. More than once, the professor next door would step in and ask him to keep it down. He spent maybe an hour explaining Euler's identity - and I shit you not, he got us to tear up, too.

We had one homework assignment. It was given about halfway through the class. We had a week to do it. And it took that entire week, in groups of two or three, five to six hours a day to do it. And we handed it in, and he didn't even grade it for like three more weeks.

When we got it back, there was a pallor over the class (what was left of it - a third of the class had dropped out). I had a 23%. I'm sure I turned gray. I went to see him - what the hell could I do? I mean, I needed to pass -

"DoNotWorryaboudit. AveragewaseighTEEEN. Yooodooverygooood."

When it came down to the final, it was one, simple, benign sheet of paper. It had one problem on it. There were absolutely no numbers on it other than (1). It started with "imagine a function..."

We had two hours. At 1:15, nobody had handed in a thing. I was just sitting there stunned, grinding through the first half. At 1:30, nobody had handed in a thing. At 1:45, he said

"Eeef...youtakezeetest choam wityou and feenishit, I veel passyou."

Nobody got up. We sat there and cranked through the fucking test. The survivors, anyway. We started with 30 people in the class. We finished with twelve.


--------------------------------------------------------------------------------

I kept maybe five textbooks from college. One of them was Ordinary Differential Equations. It was an expensive book - too expensive for me to afford. The first time I went to see him, I apologized for not having the book.

He gave me his."

Barak Hoover Obama

A very interesting article in this month's Harper's (http://www.harpers.org/archive/2009/07/0082562?redirect=1022411470, subscription required) contrasts Hoover - a deliberative, progressive, well-meaning technocrat - with Barak Obama.

The article is making the case that, although Obama is often compared with FDR, by trying to take the "middle road" and avoiding the open warfare with his detractors, he is actually emulating Herbert Hoover, and that this approach is doomed to failure.

"Franklin Roosevelt also took office imagining that he could bring all classes of Americans together in some big, mushy, cooperative scheme. Quickly disabused of this notion, he threw himself into the bumptious give-and-take of practical politics; lying, deceiving, manipulating, arraying one group after another on his side—a transit encapsulated by how, at the end of his first term, his outraged opponents were calling him a “traitor to his class” and he was gleefully inveighing against “economic royalists” and announcing, “They are unanimous in their hatred for me—and I welcome their hatred.”

Obama should not deceive himself into thinking that such interest-group politics can be banished any more than can the cycles of Wall Street. It is not too late for him to change direction and seize the radical moment at hand. But for the moment, just like another very good man, Barack Obama is moving prudently, carefully, reasonably toward disaster."

More TFS blues - adding a user

The more I use this system, the more I get a suspicion that it was not designed by developers. Or maybe we found a totally clean room bunch of developers that have never used a source control system before? (Would that be... PMs?)

I wrote previously (http://1-800-magic.blogspot.com/2009/03/adventures-in-tfs-continued.html) about the incredible amount of pain TFS is to install, and put together a simple step-by-step guide on how to install it here: http://1-800-magic.blogspot.com/2009/03/how-to-install-tfs-on-single-domain.html. (Incidentally, this is now one of the most popular articles on the blog.)

Yesterday I spent about an hour trying to figure out how to allow my daughter to use TFS server that I've set up for our home projects.

In Perforce (Microsoft uses a derivative of Perforce for quite a few of its internal projects) everything is simple: you type "p4 protect" (or "sd protect" at Microsoft), it opens a file in a notepad, and you add a line that looks like this:

write user sergey * //depot/...

And you are done.

Here's what you have to do in TFS:

(1) Add a user to a list of "licensed users". This list is not displayed by default when you navigate to the project's security settings, you have to click on a checkbox to make it display all groups.

I missed this step, and it was not mentioned on the TFS documentation page that deals with setting permissions (http://msdn.microsoft.com/en-us/library/bb558971.aspx). I performed all the magic incantations from that page, but TFS still would not connect.

And of course the error that it was showing listed security among three other options, but gave no suggestions of what might go wrong.

Eventually, a Google search on the error code led me to the words "licensed user", and then here: http://msdn.microsoft.com/en-us/library/ms404880.aspx.

(2) You have to add the user to Contributor group in your project using TFS Explorer.

This is described here: http://msdn.microsoft.com/en-us/library/bb558971.aspx.

(3) Separately, you have to grant the user access to the sharepoint site.

Also here: http://msdn.microsoft.com/en-us/library/bb558971.aspx.

(4) Separately, you have to grant the user access to the reporting portal.

And again, here: http://msdn.microsoft.com/en-us/library/bb558971.aspx.

I understand cutting features to make the deadlines, but c'mon, ladies and gentlemen of TFS, does this thing really have to be such a pain in the butt for the administrator? Especially that we're competing with Perforce where deployment is done by a single double-click?

What is your number of nines?

Ran into an interesting page today - a list of scheduled down times for Blogger: http://status.blogger.com/.

It looks like Blogger is down for roughly 10 minutes once a month (in addition to a Picasa downtime that impairs its ability to accept images).

10 minutes a month does not look like much, but it does amount to about 2 hours of downtime per year. Is two hours a year good or bad?

The system's availability is defined as the ratio of uptime to the total time:

                      MTTF
    Availability = -----------
                   MTTF + MTTR

where MTTF is the mean time between failures, and MTTR is the mean time that takes to bring the system back online. The "failure" here should be understood as a measure of system ability to process requests rather than a fault: a scheduled downtime is not a bug, but the system is not available nevertheless.

2 hours of downtime in a year yield the availability of 365.25 * 24 / (2 + 365.25 * 24) = 99.9%, or "3 nines", which puts Blogger in a category of "Well-managed" systems.

Here are the definitions of various levels of availability given in Jim Gray's famous book on transaction processing (http://www.amazon.com/Transaction-Processing-Concepts-Techniques-Management/dp/1558601902):

System type	Unavailability (min/year)	Availability	Class
Unmanaged	52560	90%	1
Managed	5256	99%	2
Well-managed	526	99.9%	3
Fault-tolerant	53	99.99%	4
High-availability	5	99.999%	5
Very-high-availability	0.5	99.9999%	6
Ultra-availability	0.05	99.99999%	7

As the Blogger's example shows, it's fairly hard to create a fault-tolerant (or above) system - you have to account for things that range from OS and software patching to the maintenance of the power equipment in the data centers.

One might think that the hardware failures and software bugs cause most of the availability problems, but it is actually the scheduled maintenance that creates majority of work, because it causes a lot of downtime. Once you figured out how to deal with the maintenance, the unavailability due to bugs is probably already taken care of by the same measures.

And at server MTTF of roughly 14 years, one should only be worrying about hardware (assuming that the failure can be detected and the job reallocated within one hour) when availability starts approaching 5 nines.

How many nines does your system have?

Navigating the Dell price labyrinth

Last week I bought a couple of big (8-core, 32GB, fast disk) boxes from Dell because they closely match the hardware that we're going to be running on in our data centers. For the speed and quality of the hardware the price ended up being very reasonable $4k/box (when bought with Microsoft discount).

While doing it, I discovered a curious thing - if you configure the box with 32GB RAM upfront, the memory comes up quite costly. If you just buy the server with 8GB default, and buy 32GB RAM separately - the peripherals section of the same Dell web site, the Dell-recommended RAM upgrade for this very workstation - the total cost is over $800 less (and you end up with 8GB of unused RAM that originally was there).

If you work at Microsoft, use this trick and watch our stock price go up 50 cents!

If, furthermore, you buy disks separately, you save another $150 or more over the price of pre-installed hard drives.

These are Microsoft-internal prices, which - for obvious reasons - I can not quote, but the problem is even bigger on the external Dell web site, because everything is even more expensive there.

Here, for example, is the price for memory - preconfigured - for Precision T7400.


You can see that the price for 32GB is $2960, and for 64GB it is a whopping $17870 (!).

Alternatively you can buy the same RAM on Newegg, so for 32GB you will by 4 8GB kits at $165-$240 for a total of $660 to $960:


Or, for 64GB you would buy 8 of these, at $420 each - $3360 - almost $15000 cheaper than on the Dell's web site!


It goes beyond RAM.

Dell wants $550 for 1TB hard drive (although they give their small business buyer a break - a 1 TB drive for the same T7400 there is "only" $430).


The prices on Newegg for 1TB hard drive range from $110-150 for a retail box to $74-$90 for OEM packaging.

Morale - if you are buying Dell computers, getting the parts on the side will save you a bundle. It is much cheaper to buy the minimal configuration, throw away the memory and the hard drive it comes with, and buy the replacement RAM and disks from Newegg (or anywhere else).

Note that the same is not true with CPUs - Dell CPUs are ~$200 more expensive than the same parts on Newegg, BUT you have to have a non-standard Dell heatsink, which - when bought separately - is very pricey. Plus replacing CPUs is not as trivial as RAM and disks.

Another interesting observation is that prices on Dell's home/small business site are often - usually - considerably less than on corporate web site. Most likely Dell uses this sales tactic to give its corporate users a "discount". Recently I bought a laptop using Microsoft EPP program, just to discover that the 7% "discount" that Dell provides simply matches the price that is available on its small business site for all.

Finally, for peripherals - docking stations and the like - it pays to check eBay. A $199 (plus shipping, handling, and tax) advanced port replicator for Latitude can be easily had there - new - for $129 (reasonable shipping, and no tax).

Overlapped I/O in Windows

One of the current puzzles that our team is dealing with is database performance. As part of our platform we're building a performance counter collection system, most of which lives in a datawarehouse-like structure in SQL server.

The specific problem we're facing is a lack of parallelism on data loads. Our usage does not quite fit standard database models, of which two are typical - OLAP (on-line analytical processing - essentially read-only database that is updated, say, nightly, contains tons of data, and is queried frequently using specific set of queries for which it was designed) and OLTP (on-line transaction processing - where the "hot" subset of the data is smaller, but is read and updated very frequently).

Our database is the worst of both worlds - it writes about 1 million rows a minute, and the reads are rather infrequent. (An argument that SQL server is not the right technology for this can be made, but this discussion is outside the context of this post).

There are two basic problems.

First is the lack of parallelism in bulk transfers on the SQL server - on an 8-core, 32GB machine with 3 10k RPM data drives (which is what $4k currently buys at Dell) the load (and index update) part is completely single-threaded. Which means that one core runs at 100% CPU, and the rest are doing nothing.

I wrote a very simple program that does all the data transforms in memory, and this part is completely parallel - the server runs at 60-70% CPU utilization - and is very fast. Unfortunately, the very last insert into SQL - Amdahl's law! - is now controls the overall performance.

The second - bigger - problem is that eventually the index no longer fits in the RAM, and due to the nature of the data it seems to be rewritten almost entirely on every upload. If I restrict SQL memory to 6GB, this covers roughly 2 hours of input, the disk starts thrashing really badly, and the load times go to hell - what initially would take only 30 seconds becomes 3 minutes.

One potential solution is to partition the database, but once we do it, it starts negatively affect the performance of our queries.

The data flow through the system is really quite small relative to the power of the hardware. The rows we're writing are only a few dozens of bytes each, and the aggregate data flow is less than 1MBps. But because the entire index is being rewritten all the time, the box keeps writing at at the rate of 20-30 MBps for several minutes.

This seemed kinda slow (although speeding it up is of course not going to be a part of the solution - we need to figure out a more global approach) - so I wanted to check what the hardware is capable of doing in terms of disk throughput, so I quickly typed up a piece of code that also makes a reasonable tutorial on how to use overlapped I/O on Windows. Hence this article.

The essential steps as as follows.

First, open the file using FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING:

HANDLE hFile = CreateFileW(szFileName, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING, NULL);

Second, you want to use DMA for the data transfer, to avoid copying, and for that the data buffer should be aligned on a sector boundary. The easiest way is to VirtualAlloc it, since this will force the buffer to be contiguous and aligned on 64k:

buffer = VirtualAlloc(NULL, bufferSize, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);

Use WriteFileEx to schedule your I/O. The hEvent part of the overlapped is not used, so a developer can use it to pass context to the completion routine. The API is quite weird this way - you'd expect that hEvent would take an event to signal on I/O completion - but that's not how it works.

overlapped.hEvent = (HANDLE)1;
overlapped.Offset = (DWORD)offset;
overlapped.OffsetHigh = (DWORD)(offset >> 32);
offset += bufferSize;
WriteFileEx(hFile, buffers, bufferSize, &overlapped, WriteFinished);

where WriteFinished gets called when the write is done:

static void CALLBACK WriteFinished(DWORD dwErrorCode,
DWORD dwNumberOfBytesTransfered,
LPOVERLAPPED lpOverlapped)
{
lpOverlapped->hEvent = (HANDLE)2;
}

Finally, once the write is scheduled, the thread must be put into alertable sleep (or wait - use the SleepEx/WaitForSingleObjectEx/WaitForMultipleObjectEx functions that take the alertable flag:

SleepEx(INFINITE, TRUE);

or

DWORD dwRes = WaitForSingleObjectEx(hSomeEvent, INFINITE, TRUE);
if (dwRes == WAIT_IO_COMPLETION)
{
// write is done
...

Obviously, you will be scheduling multiple I/Os - using an array of OVERLAPPED structures and hEvent fields inside them to keep track of what has finished and what has not is handy.

Below is the full text.

Oh, yes, and that server does ~110 MBps writes on a 10K RPM disk using the code below (and ~80 MBps writes on 7200 RPM disk), with practically zero CPU utilization.

//-----------------------------------------------------------------------
// <copyright>
// Copyright (C) Sergey Solyanik. All rights reserved.
//
// This software is in public domain and is "free as in beer". It can be
// redistributed in full or in parts for free and without any preconditions.
// </copyright>
//----------------------------------------------------------------------- 
#include <windows.h>
#include <stdio.h>

#define MAX_OUTSTANDING_WRITES 64

enum WriteProgress
{
    WriteScheduled = 1,
    WriteSucceeded = 2,
    WriteError = 3
};

static void CALLBACK WriteFinished(DWORD dwErrorCode,
                                   DWORD dwNumberOfBytesTransfered,
                                   LPOVERLAPPED lpOverlapped)
{
    if (dwErrorCode == 0)
    {
        lpOverlapped->hEvent = (HANDLE)WriteSucceeded;
        return;
    }

    wprintf(L"Error: %d\n", dwErrorCode);
    lpOverlapped->hEvent = (HANDLE)WriteError;
}

int wmain(int argc, WCHAR* argv[])
{
    if (argc != 5)
    {
        wprintf(L"Usage: writedata filename total_size chunk_size "
            L"number_of_writes\n");
        wprintf(L"Note: total_size is in megabytes\n");
        wprintf(L"      chunk_size is in bytes and must be a power of 2 "
            L"greater than 2048\n");
        wprintf(L"      number_of_writes is the number of writes "
            L"that are scheduled simultaneously\n");
        return 1;
    }

    if (GetFileAttributesW(argv[1]) != 0xffffffff)
    {
        wprintf(L"%s already exists.", argv[1]);
        return 2;
    }

    int size = _wtoi(argv[2]);
    if (size <= 0)
    {
        wprintf(L"Size must be a positive number.");
        return 3;
    }

    LARGE_INTEGER bytes;
    bytes.QuadPart = (__int64)size * 1024L * 1024L;

    int bufferSize = _wtoi(argv[3]);
    if (bufferSize <= 0)
    {
        wprintf(L"Buffer size should be a positive number.");
        return 4;
    }

    if (bufferSize & (bufferSize - 1))
    {
        wprintf(L"Buffer size must be power of 2");
        return 4;
    }

    if (bufferSize < 4096)
    {
        wprintf(L"Buffer size is too small");
        return 4;
    }

    int simwrites = _wtoi(argv[4]);
    if (simwrites <= 0)
    {
        wprintf(L"Number of simultaneous writes should be a "
            L"positive number.");
        return 5;
    }

    if (simwrites > MAX_OUTSTANDING_WRITES)
    {
        wprintf(L"Number of simultaneous writes is too large.");
        return 5;
    }

    HANDLE hFile = CreateFileW(argv[1], GENERIC_WRITE, 0, NULL,
        CREATE_ALWAYS, FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING,
        NULL);

    SetFilePointerEx(hFile, bytes, NULL, FILE_BEGIN);
    SetEndOfFile(hFile);

    OVERLAPPED overlappeds[MAX_OUTSTANDING_WRITES];
    memset(overlappeds, 0, sizeof(overlappeds));

    void *buffers[MAX_OUTSTANDING_WRITES];
    for (int i = 0 ; i < simwrites ; ++i)
    {
        buffers[i] = VirtualAlloc(NULL, bufferSize,
            MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
        memset(buffers[i], i, bufferSize);
    }

    DWORD tick = GetTickCount();

    __int64 totalScheduled = 0;
    __int64 totalWritten = 0;
    __int64 nWrites = bytes.QuadPart / bufferSize;
    int nOutstanding = 0;
    for (;;)
    {
        int nScheduled = 0;
        for (int i = 0; i < simwrites; ++i)
        {
            if ((int)overlappeds[i].hEvent == WriteScheduled)
            {
                ++nScheduled;
                continue;
            }

            if ((int)overlappeds[i].hEvent == WriteSucceeded)
            {
                totalWritten += bufferSize;
                wprintf(L"\r%I64d", totalWritten);
                memset(&overlappeds[i], 0, sizeof(OVERLAPPED));
            }

            if ((int)overlappeds[i].hEvent == WriteError)
            {
                CancelIo(hFile);
                goto finished;
            }

            if (nWrites > 0)
            {
                overlappeds[i].hEvent = (HANDLE)WriteScheduled;
                overlappeds[i].Offset = (DWORD)totalScheduled;
                overlappeds[i].OffsetHigh =
                    (DWORD)(totalScheduled >> 32);
                totalScheduled += bufferSize;
                --nWrites;
                ++nScheduled;

                if (!WriteFileEx(hFile, buffers[i], bufferSize,
                    &overlappeds[i], WriteFinished))
                {
                    DWORD dwErr = GetLastError();
                    wprintf(L"Write error %d\n", dwErr);

                    CancelIo(hFile);
                    goto finished;
                }
            }
        }

        if (nScheduled == 0)
            break;

        SleepEx(INFINITE, TRUE);
    }

finished:
    for (int i = 0 ; i < simwrites; ++i)
        VirtualFree(buffers[i], 0, MEM_RELEASE);

    CloseHandle(hFile);

    int seconds = (GetTickCount() - tick) / 1000;
    if (seconds <= 0)
        seconds = 1;

    wprintf (L" in %d seconds (%d MBps)\n", seconds,
        size / seconds);

 return 0;
}

In US religion is not...

...the opiate of the people (http://en.wikipedia.org/wiki/Opium_of_the_people) - it's Coca-Cola's corporate partner!

Prepare to believe:
http://blogs.answersingenesis.org/museum/2007/07/13/thirsty-museum-guests-choose-coke/

Canadian healthcare

Various myths regarding single-payer system in Canada
http://www.denverpost.com/opinion/ci_12523427

Coding for Kim Jong-il

You are on the plane from Seattle to Beijing when your 747 makes an emergency landing at a military landing strip in North Korea. Korean security forces quickly dispose of the crew and the passengers (the "Dear Leader" (http://en.wikipedia.org/wiki/Kim_Jong-il) likes the plane and wants to keep it for himself by faking a crash), but you are spared because they learn that you are a software developer working one of the best software companies in the world.

As you learn shortly from a personal audience with the "Dear Leader" himself, Kim Jong-il - protected by a nuclear shield - is now ready to branch into software development. His goal is to build North Korea into a technology powerhouse - with a market share as strong that of Microsoft and Google, and with employees as tightly controlled as Apple's.

So you are given a choice of potential technologies one of which you will need to build in the span of one year. At the end of the year, your work will be evaluated solely on a technical success (i.e. you are not expected to win a competition with existing products, but you are expected to build something that is competitive purely from the technical point of view).

If your code passes this evaluation, you live and potentially even reap some undisclosed benefits (the "Dear Leader" is vague on this point). If it does not, a standard process for "people who know too much" is applied to you. If you decline to cooperate, the aforementioned process is applied to you right now.

If you do agree to work on the project, you are given all possible technical means to achieve your task - an unlimited number of computers (clients and server), special dedicated fiber to the Internet, etc. You are allowed to read research papers, but not source code - including, but not limited to, any open source project. Your code must be 100% clean room implementation!

No pre-existing software infrastructure is available, other than a C++/Java/C# compiler, a text editor of your choice, and your choice of a version of Windows or Linux as an operating system.

As a first step, you get to chose the project you will be working on. The options are:
(1) C++ compiler
(2) A general-purpose database engine
(3) A general-purpose Internet search engine
(4) An operating system

Which project would you choose, and why?

Today's installer WTF

We bought my parents-in-law a Zune as a present (MSFT people get a discount at the company store). They have a pretty slow DSL connection, so downloading 131MB Zune software took almost 30 minutes.

But guess what happened next? When the setup started, it started downloading again - for another half an hour.


This begs two questions:
(1) What was in the original 130MB file - just a setup program? In 131MB???
(2) How big can a media player possibly be? Media Player Classic is less than 2MB, and VLC can play absolutely every file known to humanity in 16. Yes, Zune software also has transcoders, but I've got a whole bunch of these, too - and they are just a few megs in other packages. Why do we need a good quarter of a gigabyte?

LDAP, .NET, and Annual Review

It's that time of the year again - the annual performance review started at Microsoft today.

Annual review is an important part of every Microsoftie's career - it defines not only one's compensation, but, more importantly, reputation and mobility. It's much easier to move around the company if the review history is good. The more fun the project is, the bigger competition it attracts, and the more discriminating the hiring managers are when it comes to review history.

Good reviews are required (but, of course, not sufficient) for a great career at Microsoft.

Our team practices peer performance reviews. This is one of the Googleisms that I brought back from my year there. The theory is that peers know more than a manager about the work done by the employee, and are harder to fool, so majority of the review feedback and the ultimate review grade comes from them.

(See more here: http://1-800-magic.blogspot.com/2007/12/life-by-committee-or-management-google.html, and if you work at Microsoft and want to know more, drop me an email - I have written a whitepaper on peer review system and will be glad to share it with you.)

One of the artifacts of the peer review model is the amount of mail every manager has to send to request it - say, 5 employees times 6 reviewers per employee - that's 30 emails.

With tons of cut and paste, it took me over an hour of frantic typing last time around when we started using peer reviews during the mid-year career discussion cycle.

Then of course there is the fun of verifying that the right mail goes to the right person - individually, because you don't want to share everybody's reviewer list with everybody else, - that I didn't by mistake send a mail that is intended for one reviewer to the other, that all names in the email are correct, etc.

This time I was not about to sit and cut and paste for hours again - the problem called for a programmatic solution.

I could have probably easily put together a web site, but ensuring security a bunch of super-important personal information for a few dozen people is not exactly my idea of fun - and can't be reasonably done in under 30 minutes. The data had to stay in email.

But at least I could send the invitations automatically.

A tool was pretty quickly born that took an alias of the reviewee, and a list of aliases of the reviewers, formatted email from a template, and sent it out. In the process, I found out how to extract information from Active Directory using LDAP - I needed to get the names of the users from their aliases.

The job is surprisingly easy using DirectorySearcher class of the System.DirectoryServices namespace (you do need to add the reference to it before using).

Here's a code snippet that extracts and prints a bunch of interesting information about a user from its AD record.

Happy LDAP'ing!

//-----------------------------------------------------------------------
// <copyright>
// Copyright (C) Sergey Solyanik. All rights reserved.
//
// This software is in public domain and is "free as in beer". It can be
// redistributed in full or in parts for free and without any preconditions.
// </copyright>
//----------------------------------------------------------------------- 
namespace ldapquery
{
    using System;
    using System.Collections.Generic;
    using System.DirectoryServices;
    using System.Linq;
    using System.Text;

    /// <summary>
    /// Sample for LDAP usage in C#. Looks up a name, and then 
    /// </summary>
    class Program
    {
        /// <summary>
        /// Prints out various interesting user properties from ActiveDirectory
        /// </summary>
        /// <param name="args"></param>
        static void Main(string[] args)
        {
            if (args.Length == 0)
            {
                Console.WriteLine("Usage: ldapquery alias1 alias2...");
                return;
            }

            DirectorySearcher ds = new DirectorySearcher();

            ds.PropertiesToLoad.Add("displayName");
            ds.PropertiesToLoad.Add("givenName");
            ds.PropertiesToLoad.Add("telephoneNumber");
            ds.PropertiesToLoad.Add("mobile");
            ds.PropertiesToLoad.Add("homephone");
            ds.PropertiesToLoad.Add("mail");
            ds.PropertiesToLoad.Add("title");
            ds.PropertiesToLoad.Add("department");
            ds.PropertiesToLoad.Add("manager");

            foreach (string alias in args)
            {
                ds.Filter = "(SAMAccountName=" + alias + ")";
                SearchResult result = ds.FindOne();
                if (result == null)
                {
                    Console.Error.WriteLine("Could not resolve {0}", alias);
                    continue;
                }

                Console.WriteLine("{0}:", alias);

                if (result.Properties["displayname"].Count > 0)
                {
                    Console.WriteLine("Display name: {0}",
                        result.Properties["displayname"][0].ToString());
                }

                if (result.Properties["givenname"].Count > 0)
                {
                    Console.WriteLine("Given name: {0}",
                        result.Properties["givenname"][0].ToString());
                }

                if (result.Properties["telephonenumber"].Count > 0)
                {
                    Console.WriteLine("Office Phone: {0}",
                        result.Properties["telephonenumber"][0].ToString());
                }

                if (result.Properties["mobile"].Count > 0)
                {
                    Console.WriteLine("Mobile Phone: {0}",
                        result.Properties["mobile"][0].ToString());
                }

                if (result.Properties["homephone"].Count > 0)
                {
                    Console.WriteLine("Display name: {0}",
                        result.Properties["homephone"][0].ToString());
                }

                if (result.Properties["mail"].Count > 0)
                {
                    Console.WriteLine("Email: {0}",
                        result.Properties["mail"][0].ToString());
                }

                if (result.Properties["title"].Count > 0)
                {
                    Console.WriteLine("Title: {0}",
                        result.Properties["title"][0].ToString());
                }

                if (result.Properties["department"].Count > 0)
                {
                    Console.WriteLine("Department: {0}",
                        result.Properties["department"][0].ToString());
                }

                if (result.Properties["manager"].Count > 0)
                {
                    Console.WriteLine("Manager: {0}",
                        result.Properties["manager"][0].ToString());
                }

                Console.WriteLine();
            }
        }
    }
}

SSL certs @ GoDaddy are ~$13 a year...

...up to 10 years.

I finally bit the bullet and bought a real cert for my server at home... And VPN to my home network now works from Windows 7. Yay!

Israel threatens the US

"We want to reach an agreement with the United States on ways to advance the peace process," said a senior Jerusalem official. The U.S. stance, he said, "will stall the process and bring about tension and stagnation, which will hurt both Israel and the United States."

(Emphasis mine.)

http://www.haaretz.com/hasen/spages/1089205.html

Free web development tools for Microsoft platform

This article is about building websites using ASP.NET. "ASP.NET???" - you would exclaim in disbelief - "But this is 2009! Ruby! Rails! Python and PHP! Why would anyone even look at this boring last-century technology for building HR applications???"

There are several reasons why ASP.NET is worth more than a shrug in today's web development.

First, the tools are REALLY nice. Visual Studio absolutely rocks as a development environment.

Because of the Intellisense magic, most lines of code take only a few keystrokes to enter. Here is an example:

(1) 'c':



(2) '.':



(3) '.':



(4) '.(DownArrow':



(5) ');'

...and a line of code has been entered in roughly 8 keystrokes and under 5 seconds. When you type say a thousand lines of code per day, believe it or not, this makes a huge difference.

The debugger is fantastic - you can set breakpoint on the client and on the server, in C# and in JavaScript, and it works the way you'd expect it to work. In my days in Gmail the debugging experience was Firebug and even that involved an inordinate amount of prayer to my atheistic gods. Debugging in IE required making a pact with the devil. And as you know, Python has no debugger at all, because Pythonic code does not require debugging by definition :-).

Second, there is a lot to be said for a type-safe language.

In the world where most development is confined to 500-lines-of-Javascript-and-a-lot-of-JPEGs Facebook apps, the idea that every object is a hash map, and every squence of keystrokes is a variable makes perfect sense. Why spend time thinking about class hierarchies when we could be delighting the customers instead?

But suppose your app is 100000 lines of code. Standard software development principles - code isolation, compiler-time type checking, interfaces - become incredibly important.

How many huge applications have you seen that are written in dynamic language? I've worked on one - Gmail - which was around 250Kloc of JavaScript at the time. The only way doing anything there was possible was because of Google's internal JavaScript tools that did a lot of static code-checking at compile time, and because the v2 code base was built on object-oriented concepts - class libraries, inheritance, etc. It was very unusual - and very restricted - JavaScript for sure!

But even with all of these great tools, the life was very hard - I don't think Gmail team will be able to increase their code base very much further...

One huge advantage of ASP.NET is its ability to model an HTML page in C# - every HTML element is created as a .NET object on the server, and is then rendered into HTML. This allows your compiler to be very upfront about things that you would otherwise be finding through testing or user feedback!

For example, this is a code snippet that creates a very simple web page:

    Label l = new Label();
    l.Text = "<h1>Welcome!</h1>";
    ActivePage.Controls.Add(l);
    TextBox t = new TextBox();
    t.ID  = "textbox";
    ActivePage.Controls.Add(t);
    Button b = new Button();
    b.Text = "Submit";
    b.ID = "submit";
    b.Click += new EventHandler(clickProcessor);
    ActivePage.Controls.Add(b);

I think it was primarily because of the tools and the language that I was able to clone Google Mondrian in C# in less than a quarter of time it took Guido to write it in Python. (http://malevich.codeplex.com)

Alright, alright, you would say, but don't I have to pay Micro$oft thousands of dollars for the privilege? Ruby, PHP, Python, and MySQL are free, and have you checked the going rate for SQL Server recently?

This used to be true, and it was my constant mantra at Microsoft that we should ship at least the basic dev tools for free. It was always my opinion that if we did ship free C compiler with DOS and Windows, GNU tool chain - and, consequently, Linux - would not have existed.

Maybe I was not alone in thinking this, and better late than never, so starting around 2005 Microsoft began shipping free Express version of Visual Studio Suite, as well as SQL Server Express Edition. This articles is an introduction to these tools.

Without much ado, let's dig into free development with Microsoft tools.

First, you need a free OS. Windows 7 RC is free (for a year :-)) and is available here:
http://www.microsoft.com/Windows/Windows-7/download.aspx

On a modern DSL connection it takes about an hour to download, and another 30 minutes to install (including the updates) - at least on a virtual machine I used.


After the OS is installed, go here: http://www.microsoft.com/web/downloads/platform.aspx and download Microsoft Web Platform using "Installer 2.0 Beta". Make sure you choose everything under the Database tab, and "Visual Web Developer 2008 Express with SP1" under Tools.

Before the installation, it will ask you what authentication scheme to use for SQL Server. Select Windows Authentication.



During the installation, it will complain multiple times about "Known compatibility issues" between Win7 and various SQL Server components. Click "Continue" every time - but you will need to install SQL Server SP1 (later).

Once the installation is complete, go to Windows Updates, and click "Get updates for other Microsoft products". Install all default and optional updates (that are not language packs). Pay attention to the optional updates - the list should include SQL Server SP1. Make sure this gets in.

Your setup is now complete! From start to finish, on a decent DSL connection, it should take no more than 2.5 hours do download and install everything.

You can use free version of Visual Studio for building web applications, and a free version of SQL Server as a backend for them.

As an example of using these tools, we'll build a simple guest book app.

Our app will consists of a database that will store comment records in our guest book, and a web front-end that would allow visitors to enter comments (as well as their names and email addresses).

Let's create the database!

Our database will be really simple. It will contain records which will consists of the record key (an integer), a 50-character name, a 50-character email alias, and an unlimited comment string.

Run Microsoft SQL Server Management Studio (from Programs->Microsoft SQL Server 2008). Right-click on the list of the databases, and select "New Database...". Enter the name of the database (GuestBook), and click OK.


This creates an empty database. Now expand its tree in the left pane, and click on the table. Select "New table...". A windows that allows you to edit the database structure will open. DO NOT save the table until all the editing is done!

First, let's create an identity field - the name would be "Id", the type "int". Right-click on the field and chose "Set Primary Key".


Then, in the list of options below, make this an identity record (a unique key that identifies the record in the database).


Then create the other fields: Author and Email (both nvarchar(50)), and Comment ((nvarchar(MAX)) - all three of them NOT NULL.


Save the table (Control-S) and name it CommentRecord:


Now that we have the place to store data, we should create a security model for it. SQL Server has two security primitives that are of interest here - database users, and SQL Server logins. "A login" is a SQL equivalent of a Windows user. When you allow a user to connect to the database engine, you create a login for this user. In our case, we want to allow IIS user to connect to SQL database, so we create a login for "IIS APPPOOL\DefaultAppPool" by right-clicking on Security/Logins and selecting New... from the menu:


Logins are global to the database engines, and are a way to identify external users to SQL. The databases also have a concept of users, which are projections of logins on the individual databases (why SQL developers decided to have two entities instead of one, beats me). So we now have to create a user for the GuestBook database, and associate it with the login we just created. We will name the user "GuestBookUser". Expand the GuestBook database tree, Security, and right click on Users, and select New. Enter the logon name (IIS APPPOOL\DefaultAppPool), the user name (GuestBookUser), and check db_datareader role membership below, as follows:
.

Note that we have not granted the database user any "write" rights yet, but we expect it to be able to add the records. We will do it by creating a stored procedure (a piece of code inside our database) that would add records, and grant the GuestBookUser the right to execute this code. This way, any visitor to the web site would be able to add data, but not change anything that is already in the database.

Under the database tree, expand Programmability, and right click on New Stored Procedure. The nice thing about SQL Server 2008 is that its Management Studio supports Intellisense:


Enter the following stored procedure into the query window, and click on Execute:


Now the only thing left is to grant GuestBookUser the execute access to this stored procedure. Expand Stored Procedures under Programmability, right-click on its name (refresh - F5 - if it's not there), then select Properties, and then Permissions:


We're done! Once you do it once or twice, creating simple databases like this only takes 5 to 10 minutes.

We can now create the front-end application.

Execute Visual Web Developer 2008 Express Edition AS ADMINISTRATOR. You will need this privilege level to publish the web site.

From the File menu select New Website. Pick ASP.NET Web Site template, set the language to Visual C#, and put it in a directory called GuestBook:


We will be using LINQ to access the database from our code. LINQ To SQL makes a database accessible by generating a C# object model for it. It's actually extremely nice. You can get at the database data by executing SQL-like queries directly from C#, by writing something like this:

    var query = from cc in context.CommentRecords select cc;
    foreach (CommentRecord r in query)
    {
        Console.WriteLine("Name: {0} Email: {1} Comment: {2}",
            r.Author, r.Email, r.Comment);
    }

LINQ supports much more complicated queries, of course, with conditions and even joins.

To add the database model, right-click on the website name in Solution Explorer, and click on Add new item. Pick LINQ To SQL Classes, and change the name to GuestBook.dbml:


We now need to connect to our database. Click the Database Explorer, then Add Connection:


Pick Microsoft SQL Server:


Set the SQL Server instance as YOURMACHINENAME\SQLEXPRESS, and select GuestBook from the list of databases. Click on Test Connection to make sure everything works:


You can now see the database in the database explorer. Expand the nodes for tables and stored procedures, and drag the CommentRecord table and AddComment SP over to their places on the designer surface:


Save everything (Control-S). The database classes are ready for use!

We need an element on the form to use as a root for our HTML object model. Every HTML element we create will attach to this element. Open Default.aspx file, and convert the empty div element inside the form to asp:PlaceHolder. It needs to have properties runat="server" and id="ActivePage". Save!


Open the Default.aspx.cs file now. This is the code we're going to write - the whole thing is barely over 100 lines long, and almost 20 of that is generated as part of the template:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;

public partial class _Default : System.Web.UI.Page 
{
    //
    // This renders the page. Page_Load gets called both when
    // the page is first displayed, and also on post-backs.
    // The postback cycle initializes the page object model by running
    // the code below, then sets the state of the objects from user
    // input, then the handlers for the controls are called (in our case,
    // it is the Click handler for the button). The handlers may (and often
    // do redirect page to cause re-rendering with the new information.
    //
    protected void Page_Load(object sender, EventArgs e)
    {
        Label l = new Label();
        l.Text = "<h1>Welcome to my Guest Book!</h1>";
        ActivePage.Controls.Add(l);

        //
        // Open the connection to the database.
        //
        GuestBookDataContext context = new GuestBookDataContext(
            System.Web.Configuration.WebConfigurationManager.ConnectionStrings[
            "GuestBookConnectionString"].ConnectionString);

        Table t = new Table();
        ActivePage.Controls.Add(t);

        //
        // First, display all existing comments
        //
        var commentsQuery = from cc in context.CommentRecords select cc;
        foreach (CommentRecord record in commentsQuery)
        {
            TableRow row = new TableRow();
            t.Rows.Add(row);
            TableCell cell = new TableCell();
            row.Cells.Add(cell);
            cell.Text = Server.HtmlEncode(record.Author) + " says...";
            cell.Style.Add(HtmlTextWriterStyle.FontStyle, "italic");
            cell.Style.Add(HtmlTextWriterStyle.Color, "blue");
            cell.ColumnSpan = 2;

            row = new TableRow();
            t.Rows.Add(row);
            cell = new TableCell();
            row.Cells.Add(cell);
            cell.Text = Server.HtmlEncode(record.Comment);
            cell.ColumnSpan = 2;
        }

        context.Dispose();

        //
        // Now create the form for a user to enter comments.
        //
        TableRow author = new TableRow();
        t.Rows.Add(author);
        TableCell name = new TableCell();
        author.Cells.Add(name);

        Label authorNameLabel = new Label();
        authorNameLabel.Text = "Your name:";
        name.Controls.Add(authorNameLabel);

        TextBox authorNameInput = new TextBox();
        authorNameInput.ID = "name";
        authorNameInput.MaxLength = 50;
        name.Controls.Add(authorNameInput);

        TableCell email = new TableCell();
        author.Cells.Add(email);

        Label authorEmailLabel = new Label();
        authorEmailLabel.Text = "Email (will not show):";
        email.Controls.Add(authorEmailLabel);

        TextBox authorEmailInput = new TextBox();
        authorEmailInput.ID = "email";
        authorEmailInput.MaxLength = 50;
        email.Controls.Add(authorEmailInput);
        
        TableRow comment = new TableRow();
        t.Rows.Add(comment);

        TableCell commentCell = new TableCell();
        commentCell.ColumnSpan = 2;
        comment.Cells.Add(commentCell);

        TextBox commentInput = new TextBox();
        commentInput.Columns = 80;
        commentInput.Rows = 20;
        commentInput.TextMode = TextBoxMode.MultiLine;
        commentInput.ID = "comment";
        commentCell.Controls.Add(commentInput);

        TableRow submit = new TableRow();
        t.Rows.Add(submit);

        TableCell submitCell = new TableCell();
        submitCell.ColumnSpan = 2;
        submit.Cells.Add(submitCell);

        Button submitButton = new Button();
        submitButton.Click += new EventHandler(submitButton_click);
        submitButton.Text = "Submit comment.";
        submitCell.Controls.Add(submitButton);

        if (Page.IsPostBack)
        {
            TableRow error = new TableRow();
            t.Rows.Add(error);

            TableCell errorCell = new TableCell();
            errorCell.ColumnSpan = 2;
            errorCell.ID = "error";
            error.Cells.Add(errorCell);
        }
    }

    //
    // This is run when the user clicks on the Submit button
    //
    void submitButton_click(object source, EventArgs e)
    {
        string comment = ((TextBox)FindControl("comment")).Text;
        string name = ((TextBox)FindControl("name")).Text;
        string email = ((TextBox)FindControl("email")).Text;
        if (String.IsNullOrEmpty(comment) || String.IsNullOrEmpty(name) ||
            String.IsNullOrEmpty(email))
        {
            TableCell error = (TableCell)FindControl("error");
            Label l = new Label();
            l.Text = "You need all three values!";
            l.Style.Add(HtmlTextWriterStyle.Color, "red");
            error.Controls.Add(l);
            return;
        }

        GuestBookDataContext context = new GuestBookDataContext(
            System.Web.Configuration.WebConfigurationManager.ConnectionStrings[
            "GuestBookConnectionString"].ConnectionString);

        context.AddComment(name, email, comment);

        context.Dispose();

        //
        // This causes the page to be redrawn with the new comment in it.
        //
        Response.Redirect(Request.FilePath);
    }
}

When the code is done, you can run it directly from Visual Studio (F5), and all normal debugging primitives (breakpoints, stepping, etc) work as you'd expect them to work.

Let's deploy the web site to the real web server now. Create a directory under c:\inetpub called GuestBook. Right-click on the web site name in the Solution Explorer, and pick Copy Website option. In the dialog that opens, click on the connect button, select File System, and navigate to c:\inetpub\GuestBook. Select all the files on the left and click on the right arrow to copy them to the target directory:


Open IIS manager, navigate to Default Website, right-click on it, and pick Add Application. In the dialog box that opens enter GuestBook as the name, and c:\inetpub\GuestBook as a path to the application.


Now go to the Firewall Control Panel Applet, and open the port for IIS:


This is it!

Google I/O 2009

An interesting, if long, video: http://www.youtube.com/watch?v=S5aJAaGZIvk&feature=PlayList&p=41F4CEB92D80C4B7&index=0

Main theme is HTML 5. The consequences of it for Silverlight merit quite a bit of thought. Silverlight comes with far better tools, of course - the dev experience will be very hard to compete against. But HTML 5 will have the advantage of ubiquity.

The other very interesting demo - for me, at least - was Java support in Google App Engine. It's an Eclipse plug-in with a built-in deployment support, GWT (http://en.wikipedia.org/wiki/Google_Web_Toolkit) and server-side support all built-in. Not nearly as slick as ASP.NET, but definitely a step in that direction.

A minor thing that I found quite funny, actually, was that the main presenter on the conference was Vic Gundotra - a former marketing/evangelist guy from MSFT who is now a VP of Engineering at Google. The silliness of Google management strikes again - say what you want about Microsoft, but at least our engineering managers are real engineers :-).

In Korean news...

Absolutely hilarious piece...
http://www.kcna.co.jp/index-e.htm

Incidentally, I cannot help noticing that "Korean Central News Agency" has a Japanese domain suffix... This is what Wikipedia has to say about Internet in North Korea:

"North Korea's first Internet café opened in 2002 as a joint venture with South Korean internet company Hoonnet. It is connected via a line to China. Foreign visitors can link their computers to the Internet through international phone lines available in a few hotels in Pyongyang. In 2005 a new internet café opened in Pyongyang, connected not through China, but through the North Korean satellite link. Content is most likely filtered by North Korean government agencies.[3][4] In 2003 a joint venture called KCC Europe between businessman Jan Holterman in Berlin and the North Korean government brought the commercial Internet to North Korea. The connection is established through a satellite link from North Korea to servers located in Germany. This link ended the need to dial ISPs in China.[5]

KCC Europe is attempting to regulate the .kp country code top-level domain (ccTLD); as of 2008[update] its site (kcce.kp) and Naenara (naenara.kp) are the only known to be active in the .kp domain. Its IP address resolves not to Asia but to servers at Internet Provider Berlin (ipberlin.com) in the German capital."

http://en.wikipedia.org/wiki/Communications_in_North_Korea

No sex for iPhone users (via Reddit)

"Apple has rejected Eucalyptus, an ebook reader that facilitates downloading public domain books from Project Gutenberg, because some Victorian books mention sex."

http://www.boingboing.net/2009/05/22/apple-says-no-projec.html

Priceless! (from Reddit today)

Filed under Dear Jimmy...


http://www.reddit.com/r/politics/comments/8n75m/dear_jimmy_pic/

And the first comment:
"Dear Jamail, I hope all is well in Guantanamo, though we all know better. We're very proud that you've not lied while being tortured. It could be a disaster if you give them a reason to attack more innocent people in order to make the suffering stop.

Dad was arrested because our neighbor covets our farmland, even though it hasn't seen water in 6 months. Your little brother decided to join the Baghdad police, but became a militant when he discovered that he could make more money from bribes from the soldiers. He lost his leg when an IED went off. Your sister was raped by mercenaries and then killed to cover it up. I tried to appeal to the government, but apparently they're not subject to our laws. There's still so much death. It's hard to stay hopeful.

There's an American soldier next to me reading a note complaining about seatbelts. I wonder if they even know why they're here."

http://www.reddit.com/r/politics/comments/8n75m/dear_jimmy_pic/c09th7o

Why you should always buy memory separately

Dell, Latitude E6500, (through large enterprise portal, today): 4GB is $135


Newegg: 4GB is <$50

US health care by the numbers

• Per-capita US health care expenditures in 2002: $5,267 ("Health Care Politics and Policy in America", Patel & Rushefsky, 2006)


• Percentage by which it is higher than OECD average: 140%


• Per-capita expenditures in Germany, France, Canada, and Great Britain: $2,817, $2,736, $2,931, and $2,160.


• Percentage of health care costs that are administrative: 31%. (http://www.pnhp.org/publications/nejmadmin.pdf)


• The same in Canada: 16.7%


• Percentage of the total US health care costs last year that went to pay salary of UnitedHealth CEO: 0.14% (http://wonkroom.thinkprogress.org/2009/05/21/elizabeth-edwards-1-of-every-700-went-to-pay-salary-of-unitedhealth-ceo/)


• US health care ranking in the world, according to WHO, in 2000: 37 (http://www.photius.com/rankings/healthranks.html)


• French health care ranking: 1


• Number of developed countries ranked lower than US: 0


• US population longevity ranking in the world: 33 (tied with Cuba, http://www.usatoday.com/news/health/2007-05-18-un-life-expectancy_N.htm)


• US rank in infant mortality: 29 (tied with Slovakia and Poland, http://www.webmd.com/parenting/baby/news/20081015/infant-mortality-us-ranks-29th)


• Number of bankruptcies in US that are medical: 50% (http://www.pbs.org/moyers/journal/05222009/transcript4.html)


• The share of those where people had health insurance: 3/4

Giving back capital

Constantly in the news here is the push from the banks to return government investments - the TARP funds - to escape from the strings that were attached to that money - specifically, the limits on executive compensation.

This is being discussed as if it were the most natural thing in the world - of course bankers want to be paid more! The question of how returning cheap capital benefits the shareholders (obviously, it does not) - ostensibly, the owners of these enterpirses - is never even mentioned.

It's a really weird concept of capitalism that we have here!

http://1-800-magic.blogspot.com/2009/04/capitalism-socialism.html

Breaking change in Malevich

This will read as a daily WTF, unfortunately... but after 6 months of development, I found a really, really nasty bug in Malevich, and if you're maintaining an installation, you should read this.

Interestingly, the bug was in my face all this time, I just wasn't paying attention.

Malevich stores a bunch of timestamps in its database - one for change list itself, one for each review iteration, and one for each file version. Normally these are stored in UTC - after all, how else would you store times in a web application where users are scattered around the globe?

Well, as it turned out, all but not all. Change list time stamp was stored as local time. Of course the bug was on display on the dashboard all this time, literally in my face. But I think this was because all the rest of the dates were in UTC and therefore not very useful - I just taught myself to ignore all time stamps. So I never noticed that unlike all the rest, these ones looked correct - and they should not have!

Anyway, for quite a while Malevich fielded complaints that the time stamps are silly, and I was thinking on and off on how I might fix that. You see, .NET runtime which Malevich is built upon is great at manipulating the time, except for one thing - the web site has no idea what user's time zone is - this is by design, a privacy issue.

The client side does know what the local time offset is, so the only way to transform time correctly must lay with the client.

So this is what Malevich does - each time stamp is wrapped in a <span id='timestamp' name='timestamp> element. When the page loads, I get all the elements by the name 'timestamp', and convert them to local time.

The conversion is pretty crappy from the globalization perspective (heck, but this is *free* software :-)) - because default JavaScript time conversion functions are terrible - they produce very long strings (for example, days of the week are spelled out by some browsers).

So I compose the date/time string myself, in US format. The only thing where I try to be smart is determine if the format is 12- or 24-hours upfront and then generate the date that would use the same format. Which does not work on Chrome because it uses 24-hour format no matter what.

//-----------------------------------------------------------------------
// 
// Copyright (C) Sergey Solyanik for The Malevich Project.
//
// This file is subject to the terms and conditions of the Microsoft
// Public License (MS-PL).
// See http://www.microsoft.com/opensource/licenses.mspx#Ms-PL for
// more details.
// 
//----------------------------------------------------------------------- 
//
// Javascript module to recalculate times to browser local.
//

//
// Is time a 24-hour format or AM/PM?
//
var TIME_FORMAT_24HR = ComputeDateFormat();

//
// Recomputes all date times from UTC to local.
//
function RebuildDates()
{
    var dates = document.getElementsByName('timestamp');
    for (var i = 0; i < dates.length; ++i)
    {
        var date = new Date(dates[i].firstChild.nodeValue);

        var month = date.getMonth() + 1
        var year = date.getYear()
        var day = date.getDate()

        if (day < 10) day = '0' + day
        if (month < 10) month = '0' + month
        if (year < 1000) year += 1900

        var hour = date.getHours();
        var minute = date.getMinutes();
        var seconds = date.getSeconds();

        var suffix = '';
        if (TIME_FORMAT_24HR)
        {
            if (hour < 10) hour = '0' + hour;
        }
        else
        {
            if (hour < 12) suffix = ' AM'; else suffix = ' PM';
            if (hour == 0) hour = 12;
            if (hour > 12) hour -= 12;
            if (hour < 10) hour = '  ' + hour;
        }
        
        if (minute < 10) minute = '0' + minute;
        if (seconds < 10) seconds = '0' + seconds;

        dates[i].firstChild.nodeValue =
            [month, '/', day, '/', year, ' ', hour, ':', minute,
            ':', seconds, suffix].join('');
    }
}

//
// Checks if the date in the current locale is AM/PM or 24 hours.
//
function ComputeDateFormat()
{
    var dateString = (new Date()).toLocaleTimeString();
    if (dateString.indexOf(' AM') == -1 && dateString.indexOf(' PM') == -1)
        return true;
    return false;
}

//
// Makes RebuildDates be called on page load.
//
if (window.attachEvent)
{
    window.attachEvent('onload', RebuildDates);
}
else
{
    if (window.onload)
    {
        var curronload = window.onload;
        window.onload = function()
        {
            curronload();
            RebuildDates();
        };
    }
    else
    {
        window.onload = RebuildDates;
    }
}

Anyway, so I make this change, and now I start looking at dates very closely, and - crrrrap! - all the time stamps now look great, except the most important one - the change list time stamp!

And the worst part of course is that it's in the database! Moreover, it's data! And it's now on a few dozen servers at Microsoft alone, and I have no idea how to even find them.

Anyway, I put notices on the web site (http://malevich.codeplex.com/Wiki/View.aspx?title=special%20note%20on%20Malevich%20upgrades, and on this blog (you're reading it), and in the change list description, and on Microsoft's internal share point site for Malevich.

I modify the installer so that it detects this condition (or, rather, the right version transition), and tries to fix up the problem:

USE [CodeReview]

DECLARE @UTCDate datetime
DECLARE @LocalDate datetime
DECLARE @TimeDiff int 
  
SET @UTCDate = GETUTCDATE()
SET @LocalDate = GETDATE()
SET @TimeDiff = DATEDIFF(hh, @LocalDate, @UTCDate) 

UPDATE [dbo].[ChangeList] SET TimeStamp = DATEADD(hh, @TimeDiff, TimeStamp)

What else can I do?

Of course, the script above does not work completely, either, because it does not account for the daylight savings time (which is really, really hard to compute in T-SQL - I love human time system!), and because it assumes that local time is the same as server time, which is of course incorrect, because a bunch of teams use the tool between here and China... But it does make the time somewhat more right than it was before.

So this is the story of the worst bug in Malevich in the 6 months since it was born.

And the moral of the story... what is the moral of this story? I wrote it primarily to attract the attention of Malevich admins to the fact that they need to be thoughtful about their next upgrade. But if you have a suggestion for a witty ending, offer it in the comments section! :-)

Why do Catholic clergy not oppose gay marriage in NY?

"The state’s Roman Catholic bishops have been somewhat distracted, too, having focused their lobbying energies this session on defeating a bill that would extend the statute of limitations for victims of sexual abuse to bring civil claims, and have appeared unprepared for the battle over marriage."

http://www.nytimes.com/2009/05/20/nyregion/20marriage.html?hp

Wow! Just... wow!

Serious play (TED)

Fair warning: allocate 30 minutes upfront. You won't be able to stop once this starts!

Going-away party

"Green was convicted last week in U.S. District Court in Kentucky of murder, rape, conspiracy and obstruction of justice in connection with a 2006 rape-and-murder south of Baghdad. A jury found him guilty of raping a 14-year-old girl, then killing her and setting her body on fire to destroy evidence. Green also was found guilty of killing the girl's parents and 6-year-old sister.
...
Ruth, who is John Green's sister, noted for the jury that Green's mother is not at the trial this week. The mother is moving and had to attend a going-away party, Ruth said."

http://www.cnn.com/2009/CRIME/05/18/kentucky.iraq.murder/index.html

You know that things are hitting the bottom...

...when even New York Times can not spell...

Telescopes can see... photons? (via Reddit).

http://government.zdnet.com/?p=4765

"Thursday, the Europeans will launch two much more advanced telescopes (right) that can see photons, even those reaching back to the Big Bang. Hubble sees visual light."

I kid you not.

This is a "technical" publication...

Where are you, Robin Crusoe? Where are you? Where have you been?

Stretched on $500k/year

"Nor does he appreciate being branded as "rich" when it's far from certain he'll ever build the kind of lavish nest egg the truly wealthy enjoy, especially after the current market meltdown."

"Kelly Lynch, the owner of a commercial maintenance company in Redondo Beach, Calif., is raising two kids with her partner, Jill Fenske, on a household income of $400,000. She's saving $800 a month for the children's college fund and $4,000 a month for retirement - a number that someday might make her rich. "If I blew my money like other people, I'd feel rich," says Lynch. Her views on taxes are befitting a born entrepreneur: "I think it would be unfair if someone tried to raise my taxes," says Lynch. "I don't think people should be penalized because they earn more.""

"Even at the upper end of the HENRY group, our cover subjects, Lindsay Mayer and her husband, Zach, a Dallas attorney, feel stretched on $500,000 a year."

"Tony Molino, 50, an attorney in Rancho Palos Verdes, Calif., speaks for legions of HENRYs: "I've worked 50 to 60 hours my entire life, and I don't have a lot left over at the end of the month. I'm comfortable, but when Joe Biden talks about sucking it up, getting patriotic, and paying more taxes, I get livid.""

http://money.cnn.com/2008/10/24/magazines/fortune/tully_henrys.fortune/index.htm

Poor people, not sure if they will ever be TRULY rich.



Here's a book that shows what really means to work hard: http://www.amazon.com/Nickel-Dimed-Not-Getting-America/dp/0805063897. And it is not sitting at the desk for 2 hours more a day. It's having 3 jobs as a janitor, a waitress, a maid, and despite working 14 hours a day not having enough money to rent an apartment or go to a doctor...

To Joe the Plumbers among us...

...who vote for lower taxes for the rich because some day they expect to be rich themselves: why not buy TWO lottery tickets instead of one? The extra money from the second win will take care of the taxes!

http://www.americanprogress.org/issues/2006/04/b1579981.html

"By international standards, the United States has an unusually low level of intergenerational mobility: our parents’ income is highly predictive of our incomes as adults. Intergenerational mobility in the United States is lower than in France, Germany, Sweden, Canada, Finland, Norway and Denmark. Among high-income countries for which comparable estimates are available, only the United Kingdom had a lower rate of mobility than the United States."

Why economic and social liberalism trends are incompatible

US has two dominant parties - the economically liberal but socially conservative Republicans, and socially liberal and economically conservative Democrats.

Many in my circle - the professional class - were long yearning for the third alternative - a party that is liberal both economically and socially. Someone to represent me :-)!

Why is this not happening? Unfortunately, for a reason.

Unrestricted economic liberalism mostly benefits the upper classes.

The lower classes are less educated, more religious (church and especially Christianity historically being used as a way to keep people in their place), and therefore more socially conservative.

Since the upper classes do not have enough votes to advance their agenda, they need to coopt the lower classes to vote against the lower classes' economic interests.

This of course is done by throwing down whatever red meat issues are currently at hand - "family values", gay marriage, "security", "morality", "gun rights", etc. Most of it is either social conservatism or fear because this is what the masses respond to.

And this is why economically liberal parties are bound to be socially conservative. There is simply no way for them to attract enough followers to matter otherwise.

Which is why you have such a high incidents of hypocrisy among Republican leaders - from Rush Limbaugh abusing drugs while preaching death to (other) drug addicts, to Larry Craig's restroom incident. They don't really give a flying f*ck about morals (or anything else they sell - gun rights, security, etc). They are just using this to get more idiots vote for them.

What does this say about the rich? Not much, unfortunately (I cannot bring myself to call Wall Street bankers and automaker CEOs "the elites" which to me means "the best"). By trying to extract the last penny while stirring the worst instincts in the society, they retard the progress of civilization.

Yes, they are rich, but as a result time moves slower for everybody. But you know what? The quality of life and the average longevity of the middle class in the age of antibiotics turns out to be better than that of the royalty in the Middle Ages.

Trading stem cell research for a few thousand bucks in taxes is not smart, and proves once again that IQ and compensation in today's America are unfortunately not correlated.

And this is usually the beginning of the end...

Why is it socially acceptable to be a conservative?

You generally don't see many people go around and say "I am a fascist!" or "I am a racist!". And yet there are plenty of people who proudly declare themselves to be conservative.

Let's take a brief look at the history.

Spanish Inquisition's goal was to suppress heresy - or, in other words, preserve traditional religious beliefs. It was a conservative movement that resulted in executions of thousands of people.

US itself was founded by progressives - conservatives were known as Loyalists at the time.

Over the last couple hundred years conservatives, standing athwart history (in the words of William Buckley), have opposed most of the civilization advances that we now take for granted:

• emancipation of slaves
• women's suffrage
• 8 hour workday
• limitations on child labour
• desegregation and racial equality
• Social Security
• Medicare

And you don't have to go too far back in time - it is probably safe to say that in 1960s most conservatives supported segregation, and that in 1920s most of them were against women's suffrage. You just cannot say the same about liberals.


Today, history is repeating itself with conservative attitudes towards everything from torture and gay rights to healthcare reform.

So why is the conservative brand still acceptable in the mainstream society after all of this? What am I missing?

The future is now!

Two links:

http://1-800-magic.blogspot.com/2007/12/windows-8-or-9-or-10.html

...and...

http://www.withinwindows.com/2009/04/24/secret-no-more-revealing-windows-xp-mode-for-windows-7/

A whole version earlier than I predicted.

Israel to US: shut up and do what I say (via Reddit)

"The Obama Administration will put forth new peace initiatives only if Israel wants it to, said Foreign Minister Avigdor Lieberman in his first comprehensive interview on foreign policy since taking office.

"Believe me, America accepts all our decisions," Lieberman told the Russian daily Moskovskiy Komosolets."

http://www.haaretz.com/hasen/spages/1080097.html

Dell Latitude: 8 hours on a single charge!

Tonight (by accident) I left my laptop unplugged and in a state where it could not go to sleep automatically. In the morning I found the laptop with its battery completely depleted, but still running!

Kudos to power management systems in Core 2 Duo and Windows Server 2008!

Laissez-faire (via Reddit)

"In Bucharest, you can buy a young girl for 8,000 euros. I mean buy them as in you own them forever or until you sell them to someone else. Most of them are sold by their parents in Moldova when they are little children. As soon as they’re bought, the owner brands them. This is normally a tattoo of his name."

http://www.viceland.com/int/v13n9/htdocs/slavery1.php?country=uk

While this sounds (reads?) insane, it is not really that unusual for an unregulated market to produce this phenomenon. It was not all this different in US in 1890s, and in Russia in 1990s - just the percentage of population is smaller.

Do laws have priority over money, or does money have priority over laws?

The Lifestyle

"It is a tricky situation in which some Americans find themselves after a long boom: They are by no means struggling, compared with the 98% of Americans who make far less, but depending on where they live and the lifestyle choices they have made, they don't necessarily feel rich, either."

http://finance.yahoo.com/focus-retirement/article/106934/Wealth-Less-Effect-Earning-Well-Feeling-Otherwise

Oh, yes, the life style choices... Take me, for instance. My personal choice, as anyone who knows me well will confirm, is to fly my own 747. I love to travel, and the fact that I have to wait in airports, sleep in a chair on long flights, and stand in long lines is simply unacceptable. It makes me feel positively broke. Middle class you say? As far as I am concerned, I am a pauper.

Now, Obama thinks that we are rich and wants to tax our hard earned money (that is, the amount above $250k) at 3% more. Which means that if we were to make, say, $350k next year, we'd be forking over $3000 more to the government in taxes. Instead of buying one of these: http://1-800-magic.blogspot.com/2009/04/computer-porn.html, which my chosen lifestyle dictates that I have. Talk about socialism!

Malevich has setup!

I've been working on Malevich - my code review system - on and off for almost 5 months now.

It turned out to be very successful - my team was using it for all of our first milestone, and was very happy about it (we're now pushing 1500 change lists that went through the system). Many other Microsoft teams adopted it since, to the tune of at least a couple hundred people - and these are just the people who contacted me.

The biggest problem so far was the setup. The deployment was hard - it required Visual Studio, a service pack, an expansion pack, and tons of manual work. A lot of errors were made, and in many cases I had to drop by and set things right afterwards.

Not any more! The first official build is out, and it has a real setup! The setup checks the dependencies, deploys the database, publishes the web site, and configures notifier automatically, no Visual Studio is required!

So if you wanted to try Malevich but were intimidated by the complexity of installation - the wait is over. You still do have to install Server 2008 with IIS and SQL Server 2008, but the rest is done for you.

Just get the MSI here: http://malevich.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=26374

Happy code reviewing!

How is this not treason? (via Reddit)

"Rep. Jane Harman , the California Democrat with a longtime involvement in intelligence issues, was overheard on an NSA wiretap telling a suspected Israeli agent that she would lobby the Justice Department reduce espionage-related charges against two officials of the American Israeli Public Affairs Committee, the most powerful pro-Israel organization in Washington."

...

"In exchange for Harman’s help, the sources said, the suspected Israeli agent pledged to help lobby Nancy Pelosi, D-Calif., then-House minority leader, to appoint Harman chair of the Intelligence Committee after the 2006 elections, which the Democrats were heavily favored to win."

http://www.cqpolitics.com/wmspage.cfm?docID=hsnews-000003098436

The new government

"As far as the most important matter is concerned, there is complete unanimity. Liberman, Netanyahu, Barak, Ellie Yishai of Shas and Danny Hershkovitz of the “Jewish Home” party are in total agreement about the Palestinians. All of them agree on the need to prevent the establishment of a real Palestinian state. All of them agree not to talk with Hamas. All of them support the settlement enterprise. During Barak’s stint as Prime Minister, the settlements grew even faster than during Netanyahu’s tenure. Liberman is himself a settler, Hershkovitz’s party represents the settlers. All of them believe that there is no need for peace, that peace is bad for us. (After all, it was Barak, not Netanyahu or Liberman, who coined the phrase “We Have No Partner for Peace”.)"

http://zope.gush-shalom.org/home/en/channels/avnery/1238277190

Computer porn

2x	Intel Xeon E5410 Harpertown 2.33GHz 12MB L2 Cache LGA 771 80W Quad-Core Processor	http://www.newegg.com/Product/Product.aspx?Item=N82E16819117150	$590
1x	SUPERMICRO MBD-X7DCL-I Dual LGA 771 Intel 5100 ATX Server Motherboard	http://www.newegg.com/Product/Product.aspx?Item=N82E16813182146	$300
6x	Kingston 4GB 240-Pin DDR2 SDRAM ECC Registered DDR2 667 (PC2 5300) Server Memory	http://www.newegg.com/Product/Product.aspx?Item=N82E16820134835	$504
5x	Western Digital VelociRaptor WD1500HLFS 150GB 10000 RPM 16MB Cache SATA 3.0Gb/s 3.5" Hard Drive	http://www.newegg.com/Product/Product.aspx?Item=N82E16822136296	$900
1x	Athena Power AP-P4ATX85FE 20+4Pin 850W Single ATX12V & EPS12V Power Supply	http://www.newegg.com/Product/Product.aspx?Item=N82E16817104126	$155
1x	SILVERSTONE KUBLAI Series KL03-B Black 2.5mm aluminum front door, 0.8mm SECC body ATX Full Tower Computer Case	http://www.newegg.com/Product/Product.aspx?Item=N82E16811163100	$155

...yields an 8-core, 24GB monster capable of hosting 10 VMs comfortably for $2600.

Capitalism, socialism

Wikipedia defines capitalism thus:

"Capitalism is an economic system in which wealth, and the means of producing wealth, are privately owned and controlled rather than state owned and controlled. Through capitalism, the land, labor, and capital are owned, operated, and traded by private individuals either singly or jointly, and investments, distribution, income, production, pricing and supply of goods, commodities and services are determined by voluntary private decision in a market economy. A distinguishing feature of capitalism is that each person owns his or her own labor and therefore is allowed to sell the use of it to employers." (http://en.wikipedia.org/wiki/Capitalism).

The definition puts "ownership" and "control" of the capital in the same sentence, but does not stipulate that it is the same individual that both owns and controls the resources.

And therein lies our problem.

Commonly accepted theory of why capitalism is preferable to socialism goes like this: the owners generally make better decisions about allocating the resources than non-owners. When the owners do make sub-optimal decision, they lose their ownership stake to the smarter and more responsible new owners.

So far, so good. Except this is not at all the system that we have in the United States. In the US, the majority of the capital is owned by one set of people - the shareholders - and managed by an entirely different set of people - the corporate management.

The interests of these groups of people are not at all aligned (http://1-800-magic.blogspot.com/2007/12/risk-vs-reward.html), and, what's worse, neither group is truly interested in the long-term success of the business.

As a result, while the executives in younger companies which are still managed by the original owners do often tend to do the right thing, I cannot call the behavior of the managerial class in most American enterprises anything other than looting.

This approach to management has been most visible on Wall Street, but look at any older US enterprise, and you will see the same looting of company resources everywhere, from car manufacturers to steel mills to utility companies.

Comparing to the old Soviet days, the system as we have it right now may actually be worse. At least the bureaucrats in Soviet Union were not paying themselves multimillion dollar bonuses and did not zip around in private planes as the economy crumbled.

Discounting risk

I wrote this article in December 2007 and it describes perfectly the situation we're in now: http://1-800-magic.blogspot.com/2007/12/risk-vs-reward.html

If you had only two news articles to read this week (via Reddit)...

...I would recommend "Ten principles for a Black Swan-proof world" by Nicolas Taleb (best known for his book "The Black Swan: The Impact of the Highly Improbable" about underpricing risk) and The dark side of Dubai. This last one is absolutely surreal. Make sure you read it at least through "The Lifestyle" chapter. It is really, truly, absolutely unbelievable.

We need this guy back!

http://images.pcworld.com/news/graphics/162166-billgatesFINAL2_original.jpg

Visual Studio SP1 installer blues

I have but one question - what is taking it so long?

It takes 1.5 hours on my 2.5GHz, 4GB Core 2 Duo laptop running Windows Server 2008.

In 1.5 hours one could, at a typical speed of my laptop:
- Write 263GB of data to disk, at 50MBps
- Transfer 131GB of data over the network, at 200Mbps
- Copy 15 TB (!) of memory (on 800Mhz bus)
- Execute a whopping 26 trillion instructions (2 cores retiring 2 instructions per core per clock).

WHAT DOES THIS THING DO???

Weather plays April Fool's in Seattle

It's April 1st, and it's SNOWING! Here's what it looks like from our window.

The Lord of the Rings and Atlas Shrugged (via Reddit)

"There are two novels that can change a bookish fourteen-year old’s life: The Lord of the Rings and Atlas Shrugged. One is a childish fantasy that often engenders a lifelong obsession with its unbelievable heroes, leading to an emotionally stunted, socially crippled adulthood, unable to deal with the real world. The other, of course, involves orcs."

http://www.amptoons.com/blog/archives/2009/03/29/quote-8/

Justice and accountability, the American way (via Reddit)

1. The hospital makes a medical error; patient dies.
2. The hospital falsifies the medical record, including the testimony of the doctor given during the investigation.
3. The doctor who gave the testimony produces the original copy.
4. The hospital settles out of the court. The article does not say this, but I bet that the people who were involved in hiding the evidence probably still work for that hospital.

“There isn’t going to be a trial,” he said. “The hospital offered the patient’s family an 8-figure settlement, and they have accepted.”

http://open.salon.com/blog/amytuteurmd/2009/03/30/they_killed_my_patient_then_they_tried_to_hide_it

Malevich hits 1000th code review!

We've been using it for 3 months now in our team of roughly 45 people - 20 devs and 25 testers. This week we've had 1000th code review completed.

Here are a few stats, as of right now:

1008 change lists
17081 file versions
2863 review iterations
7851 comments

If you work at Microsoft and would like to try Malevich, I've set up a test change list here: http://sergeysomsg2/Malevich/default.aspx?cid=299. You can add yourself as a reviewer, leave comments in files, and submit review iterations.

If you're outside Microsoft and want to try it, the source and installation instructions are here: http://www.codeplex.com/Malevich.

Welcome to the ultimate in banana republics! (via Reddit)

Written by a former chief economist at IMF.

"Typically, these countries are in a desperate economic situation for one simple reason—the powerful elites within them overreached in good times and took too many risks. Emerging-market governments and their private-sector allies commonly form a tight-knit—and, most of the time, genteel—oligarchy, running the country rather like a profit-seeking company in which they are the controlling shareholders. When a country like Indonesia or South Korea or Russia grows, so do the ambitions of its captains of industry. As masters of their mini-universe, these people make some investments that clearly benefit the broader economy, but they also start making bigger and riskier bets. They reckon — correctly, in most cases — that their political connections will allow them to push onto the government any substantial problems that arise."

(emphasis mine)

http://www.theatlantic.com/doc/print/200905/imf-advice

Empire strikes forward

It seems that vast majority of Star Wars technology was designed and built by the marketing department. Here are a few engineering suggestions that could help the bad guys win in Epsiodes -3 to 0...

1) Use firearms. Blasters are slow, not very powerful, and easily deflected by a light saber. An AK-47 would be far, far more effective. And don't forget the training! Despite the advantage in fire power, the storm troopers clearly can't shoot straight. Take them out to a range once in a while, and see how much more effective they become only after a couple training sessions!

2) If you use droids to fight your wars, build the best kind - do not waste time and money on anything else. It looks like the Destroyer model is more effective than50 units of the other kind (and even more effective than a Jedi). I doubt it could be more than 3 times more expensive. Why not just build Destroyers?

3) And while we are on this subject, let's have droids use more effective communication channels than human speech when they talk to each other. The same goes for all interfaces between the droids and everything else - spaceships, vehicles, and weapons. There is no reason while a droid should press a button to fire a weapon - a wireless interface would control it far more efficiently. Also, do invest in computerized targeting for your weapons. If your droids can already see, making them very accurate shooters is a relatively trivial task - ask any engineer.

4) Missiles seem to be an extremely effective weapon in Star Wars, except that there are very few of them. Why not build more of them and launch, say, 20 or 30 at a time? I doubt the Jedi, who have visible trouble dealing with one or two, would be able to escape ten. Also, consider building missiles that are FASTER than the star fighters they attack.

5) Consider using encryption. It looks like anyone can plug into the system anywhere and take control of everything in the space ships. This problem has been solved years ago! It really is not that hard.

6) Pack animals are slow, can't carry much, and are very, very hard to maintain. They need food, living compartments, etc. Consider eliminating them from the army in favor of mechanized transporters. Animals are not very effective in executions either - this has been proven many, many times. Instead of staging elaborate shows with giant predators that always fail to kill the prisoners, just shoot your captives, and put their heads on a stick.

7) I cannot overemphasize the importance of conventional firearms. If the probe sent to assassinate Princess Amidala had used a regular rifle instead of the poisonous centipedes, the subsequent events might have taken a very different turn. And in space, consider using nuclear weapons. The laser guns you currenlty mount on your ships are massively underpowered.

8) Did you know that space ships do not need wings to fly in space? Once you get entirely comfortable with this (yes, I do know it's extremely hard, given that they even make fying noises while moving through vacuum, but making this leap of faith might be crucial to your survival - do it!) you can start using very different design paradigms - like, for example, minimizing the surface area so your spacecraft is easier to protect and harder to target.

9) On the subject of spaceship designs - there must be something in the way you build them that makes them explode after they are hit. Unfortunately, this applies to all other vehicles as well. While this generates impressive visuals, this design point leads to unnecessary casualties among your troops. Consider enclosing whatever it is that explores in extra layers of protection. You have already solved this for your small weapons, which do not seem to have the same problem. Why not use similar design everywhere?

10) This is more of a tactical rather than an engineering advice, but I will give it anyway. Instead of deploying the Walkers and other weird ground assault vehicles, consider attacking from the air to suppress the adversary's ground troops.

Follow these simple steps, and you will be invincible - at least as long as the Good Guys(TM) are controlled by the marketing people.

Free as in beer!

The GNU connotation of software as a form of speech never made any sense to me.

"Free speech" means the freedom of a person to express himself or herself without a fear of persecution. It does not mean the freedom to use the results of this expression by everyone.

Quite the contrary - the literary works by the greatest writers are the results of exercising free speech. But rip "Harry Potter", and you will - deservedly - find yourself slapped with a lawsuit, a fine, and maybe a jail sentence.

The free software movement has produced a lot of great things - from GNU development tools to Linux, BSD, Apache, and Firefox. It has also produced legions of activists that bring in the level of politics, acrimony, and fanboyism that is unbecoming to honorable competition in an area that is as much a science as it is an art.

By doing so they actually take the freedom away from people who chose to distribute the results of their work through channels that are different from the ones favored by the Free Software Foundation.

So here's my variant of "free software". The code distributed on this blog is free - free as in beer. You can redistribute it in full or in parts, with or without source code. You can use it in any derivative work without giving me any credit.

//-----------------------------------------------------------------------
// <copyright>
// Copyright (C) Sergey Solyanik.
//
// This software is in public domain and is "free as in beer". It can be
// redistributed in full or in parts for free and without any preconditions.
// </copyright>
//----------------------------------------------------------------------- 

De-gnoming your blog

My blog has been filling up with spam over the last few months - the nasty, long messages that pollute the comment space and are a pain to delete because the bastards were programmatically sticking them in tens and tens of posts.

Luckily, Google has APIs to Blogger, and better yet, they even have C# libraries than make coding against these API easy. I didn't even have to dust off Eclipse!

So an hour later I had the program that allowed me to clean up the spam quickly.

The code sucks in the entire blog (including the abstracts of posts and comments) and allows one to quickly search and delete spam. Type 'help' on the command prompt, and it will explain the usage.

A few convenient shortcuts:

List or delete all comments from someone:
    list|delete comments by name
e.g.
    list comments by peter.w

List or delete all comments that have the same author as well as the text:

    list|delete comments like sample_comment_uri
e.g.
    list comments like http://www.blogger.com/feeds/3554166144204741789/3010398536200323405/comments/default/4945114575786176865

List or delete all comments with a phrase in title:
    list|delete comments with text_string
e.g.
    list comments with wow gold

List comments in a blog or a message:
    list comments in blog_name|first_words_of_post_title
e.g.
    list comments in Back to Microsoft
    list comments in 1-800-MAGIC

If you're one of the truly insane types that would run code compiled by a random guy from the Internet, the built version is here: http://www.solyanik.com/drop/BlogDeSpammer.zip. Otherwise, the code is below, free as in beer!

Incidentally, it might make a good primer on how to retrieve and manipulate Blogger posts and comments programmatically.

//-----------------------------------------------------------------------
// <copyright>
// Copyright (C) Sergey Solyanik.
//
// This software is in public domain and is "free as in beer". It can be
// redistributed in full or in parts for free and without any preconditions.
// </copyright>
//----------------------------------------------------------------------- 
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Google.GData.Client;

namespace BlogDeSpammer
{
    /// <summary>
    /// Comment class. Just a trivial holder of essential comment data.
    /// </summary>
    class Comment
    {
        public string Text;
        public string Author;
        public string EditUri;

        public Comment(string text, string author, string editUri)
        {
            Text = text;
            Author = author;
            EditUri = editUri;
        }
    }

    /// <summary>
    /// Post class. Just a trivial holder of essential post data.
    /// </summary>
    class Post
    {
        public string Title;
        public string CommentsFeed;
        public ICollection<Comment> Comments;

        public Post(string title, string commentsFeed)
        {
            Title = title;
            CommentsFeed = commentsFeed;
            Comments = null;
        }
    }

    /// <summary>
    /// Blog class. Just a trivial holder of essential blog data.
    /// </summary>
    class Blog
    {
        public string Name;
        public string Feed;
        public ICollection<Post> Posts;
        public Blog(string name, string feed)
        {
            Name = name;
            Feed = feed;
            Posts = null;
        }
    }

    /// <summary>
    /// Main functionality.
    /// </summary>
    class Program
    {
        /// <summary>
        /// Gets collection of blogs belonging to the user.
        /// </summary>
        /// <param name="blogger"> Blogger service. Must be initialized with the parameters.
        /// </param>
        /// <returns> Collection of blogs. </returns>
        private static ICollection<Blog> GetBlogs(Service blogger)
        {
            List<Blog> blogs = new List<Blog>();

            FeedQuery query = new FeedQuery();
            query.Uri = new Uri("http://www.blogger.com/feeds/default/blogs");
            AtomFeed results = blogger.Query(query);
            while (results != null && results.Entries.Count > 0)
            {
                foreach (AtomEntry entry in results.Entries)
                {
                    string blogFeedLink = null;
                    foreach (AtomLink link in entry.Links)
                    {
                        if (BaseNameTable.ServiceFeed.Equals(link.Rel))
                        {
                            blogFeedLink = link.HRef.ToString();
                            break;
                        }
                    }
                    blogs.Add(new Blog(entry.Title.Text, blogFeedLink));
                }

                if (results.NextChunk == null)
                    break;

                query.Uri = new Uri(results.NextChunk);
                results = blogger.Query(query);
            }

            return blogs;
        }

        /// <summary>
        /// Retrieves all the posts for the blog.
        /// </summary>
        /// <param name="blogger"></param>
        /// <param name="blog"></param>
        /// <returns> Collection of posts. </returns>
        private static ICollection<Post> GetPosts(Service blogger, Blog blog)
        {
            List<Post> posts = new List<Post>();

            FeedQuery query = new FeedQuery();
            query.Uri = new Uri(blog.Feed);
            AtomFeed results = blogger.Query(query);
            while (results != null && results.Entries.Count > 0)
            {
                foreach (AtomEntry entry in results.Entries)
                {
                    string commentsFeedLink = null;
                    foreach (AtomLink link in entry.Links)
                    {
                        if ("replies".Equals(link.Rel) &&
                            "application/atom+xml".Equals(link.Type))
                        {
                            commentsFeedLink = link.HRef.ToString();
                            break;
                        }
                    }
                    Console.Write(".");
                    if (commentsFeedLink != null)
                        posts.Add(new Post(entry.Title.Text, commentsFeedLink));
                }

                if (results.NextChunk == null)
                    break;

                query.Uri = new Uri(results.NextChunk);
                results = blogger.Query(query);
            }

            return posts;
        }

        /// <summary>
        /// Gets comments for the given post.
        /// </summary>
        /// <param name="blogger"> Blogger service, initialized with credentials. </param>
        /// <param name="post"> The post. </param>
        /// <returns> Collection of credentials. </returns>
        private static ICollection<Comment> GetComments(Service blogger, Post post)
        {
            List<Comment> comments = new List<Comment>();

            FeedQuery query = new FeedQuery();
            query.Uri = new Uri(post.CommentsFeed);
            AtomFeed results = blogger.Query(query);
            while (results != null && results.Entries.Count > 0)
            {
                foreach (AtomEntry entry in results.Entries)
                {
                    Console.Write(".");
                    comments.Add(new Comment(entry.Title.Text, entry.Authors[0].Name,
                        entry.EditUri.ToString()));
                }

                if (results.NextChunk == null)
                    break;

                query.Uri = new Uri(results.NextChunk);
                results = blogger.Query(query);
            }
            return comments;
        }

        /// <summary>
        /// Main. Does all the work.
        /// </summary>
        /// <param name="args"> Command line parameters. </param>
        static void Main(string[] args)
        {
            if (args.Length != 2)
            {
                Console.WriteLine("Usage: blogdespammer username password");
                return;
            }

            string username = args[0];
            string password = args[1];

            Console.Write("Loading");

            Service blogger = new Service("blogger", "BlogDeSpammer");
            blogger.Credentials = new GDataCredentials(username, password);

            ICollection<Blog> blogs = GetBlogs(blogger);
            foreach (Blog blog in blogs)
            {
                blog.Posts = GetPosts(blogger, blog);
                foreach (Post post in blog.Posts)
                    post.Comments = GetComments(blogger, post);
            }

            Console.WriteLine("\n\nType 'help' for help, 'exit' to quit.");
            for (; ; )
            {
                Console.Write("> ");
                string input = Console.ReadLine();
                if (input.Equals("exit", StringComparison.CurrentCultureIgnoreCase))
                    break;

                if (input.Equals("help", StringComparison.CurrentCultureIgnoreCase))
                {
                    Console.WriteLine(
                        "Type:\n\texit\n\t\t-- to quit.\n\thelp\n\t\t--to get this text.\n");
                    Console.WriteLine(
                        "\tlist blogs\n\t\t-- to list available blogs.\n");
                    Console.WriteLine(
                        "\tlist posts in blog\n\t\t-- to list posts in a blog.\n");
                    Console.WriteLine(
                        "\tlist comments in blog\n\t\t-- to list comments in a blog.\n");
                    Console.WriteLine(
                        "\tlist comments in post\n\t\t-- to list comments in a post.\n");
                    Console.WriteLine(
                        "\tlist comments by author\n\t\t-- to list comments by " +
                        "a particular author.\n");
                    Console.WriteLine(
                        "\tlist comments like commenturi\n\t\t-- to list comments " +
                        "identical to a comment.\n");
                    Console.WriteLine(
                        "\tlist comments with text\n\t\t-- to list comments " +
                        "containing text segment.\n");
                    Console.WriteLine(
                        "\tdelete comments by author\n\t\t-- to delete all " +
                        "comments by a particular author.\n");
                    Console.WriteLine(
                        "\tdelete comments like commenturi\n\t\t-- to delete " +
                        "all comments that are the same as a " +
                        "comment with this URI.\n");
                    Console.WriteLine(
                        "\tdelete comments with text\n\t\t-- to list comments " +
                        "containing text segment.\n");
                    Console.WriteLine("Names above could be either text names or URIs.");
                    continue;
                }

                if (input.Equals("list blogs", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    foreach (Blog blog in blogs)
                        Console.WriteLine("{0} ({1})", blog.Name, blog.Feed);
                    continue;
                }

                if (input.StartsWith("list posts in ", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    string blogname = input.Substring(14);
                    foreach (Blog blog in blogs)
                    {
                        if (blog.Name.StartsWith(blogname) ||
                            blog.Feed.Equals(blogname))
                        {
                            foreach (Post post in blog.Posts)
                                Console.WriteLine("{0} ({1})", post.Title, 
                                    post.CommentsFeed);
                        }
                    }
                    continue;
                }

                if (input.StartsWith("list comments in ", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    string name = input.Substring(17);
                    foreach (Blog blog in blogs)
                    {
                        if (blog.Name.StartsWith(name) || blog.Feed.Equals(name))
                        {
                            foreach (Post post in blog.Posts)
                            {
                                Console.WriteLine("{0} {1}", post.Title, 
                                    post.CommentsFeed);
                                foreach (Comment comment in post.Comments)
                                    Console.WriteLine("-- {0} ({1})\n{2}\n", 
                                        comment.Author, comment.EditUri,
                                        comment.Text);
                            }
                            break;
                        }

                        foreach (Post post in blog.Posts)
                        {
                            if (post.Title.StartsWith(name) || 
                                post.CommentsFeed.Equals(name))
                            {
                                Console.WriteLine("{0} {1}", post.Title, 
                                    post.CommentsFeed);
                                foreach (Comment comment in post.Comments)
                                    Console.WriteLine("-- {0} ({1})\n{2}\n", 
                                        comment.Author, comment.EditUri,
                                        comment.Text);
                            }
                        }
                    }
                    continue;
                }

                if (input.StartsWith("list comments by ", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    string author = input.Substring(17);
                    foreach (Blog blog in blogs)
                    {
                        foreach (Post post in blog.Posts)
                        {
                            foreach (Comment comment in post.Comments)
                            {
                                if (comment.Author.Equals(author, 
                                    StringComparison.CurrentCultureIgnoreCase))
                                {
                                    Console.WriteLine("{0} : {1}",
                                        blog.Name, post.Title);
                                    Console.WriteLine("-- {0} ({1})\n{2}\n",
                                        comment.Author, comment.EditUri,
                                        comment.Text);
                                }
                            }
                        }
                    }
                    continue;
                }

                if (input.StartsWith("list comments with ", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    string text = input.Substring(19);
                    foreach (Blog blog in blogs)
                    {
                        foreach (Post post in blog.Posts)
                        {
                            foreach (Comment comment in post.Comments)
                            {
                                if (comment.Author.Contains(text) || 
                                    comment.Text.Contains(text))
                                {
                                    Console.WriteLine("{0} : {1}",
                                        blog.Name, post.Title);
                                    Console.WriteLine("-- {0} ({1})\n{2}\n",
                                        comment.Author, comment.EditUri,
                                        comment.Text);
                                }
                            }
                        }
                    }
                    continue;
                }

                if (input.StartsWith("list comments like ", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    string prototypeUri = input.Substring(19);
                    Comment prototypeComment = null;

                    foreach (Blog blog in blogs)
                    {
                        foreach (Post post in blog.Posts)
                        {
                            foreach (Comment comment in post.Comments)
                            {
                                if (prototypeUri.Equals(comment.EditUri))
                                {
                                    prototypeComment = comment;
                                    goto found;
                                }
                            }
                        }
                    }
                    continue;

                found:
                    foreach (Blog blog in blogs)
                    {
                        foreach (Post post in blog.Posts)
                        {
                            foreach (Comment comment in post.Comments)
                            {
                                if (comment.Author.Equals(prototypeComment.Author,
                                    StringComparison.CurrentCultureIgnoreCase) &&
                                    comment.Text.Equals(prototypeComment.Text))
                                {
                                    Console.WriteLine("{0} : {1}", blog.Name, post.Title);
                                    Console.WriteLine("-- {0} ({1})\n{2}\n", 
                                        comment.Author, comment.EditUri,
                                        comment.Text);
                                }
                            }
                        }
                    }
                    continue;
                }

                if (input.StartsWith("delete comments by ", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    string author = input.Substring(19);

                    foreach (Blog blog in blogs)
                    {
                        foreach (Post post in blog.Posts)
                        {
                            List<Comment> deleted = new List<Comment>();

                            foreach (Comment comment in post.Comments)
                            {
                                if (comment.Author.Equals(author, 
                                    StringComparison.CurrentCultureIgnoreCase))
                                {
                                    Console.WriteLine("{0} : {1}", blog.Name, post.Title);
                                    Console.WriteLine("-- {0} ({1})\n{2}\n", 
                                        comment.Author, comment.EditUri,
                                        comment.Text);
                                    blogger.Delete(new Uri(comment.EditUri));
                                    deleted.Add(comment);
                                }
                            }

                            foreach (Comment comment in deleted)
                                post.Comments.Remove(comment);
                        }
                    }
                    continue;
                }

                if (input.StartsWith("delete comments like ", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    string prototypeUri = input.Substring(21);
                    Comment prototypeComment = null;

                    foreach (Blog blog in blogs)
                    {
                        foreach (Post post in blog.Posts)
                        {
                            foreach (Comment comment in post.Comments)
                            {
                                if (prototypeUri.Equals(comment.EditUri))
                                {
                                    prototypeComment = comment;
                                    goto found;
                                }
                            }
                        }
                    }
                    continue;

                found:
                    foreach (Blog blog in blogs)
                    {
                        foreach (Post post in blog.Posts)
                        {
                            List<Comment> deleted = new List<Comment>();

                            foreach (Comment comment in post.Comments)
                            {
                                if (comment.Author.Equals(prototypeComment.Author,
                                    StringComparison.CurrentCultureIgnoreCase) &&
                                    comment.Text.Equals(prototypeComment.Text))
                                {
                                    blogger.Delete(new Uri(comment.EditUri));
                                    deleted.Add(comment);
                                    Console.WriteLine("{0} : {1}", blog.Name, post.Title);
                                    Console.WriteLine("-- {0} ({1})\n{2}\n", 
                                        comment.Author, comment.EditUri,
                                        comment.Text);
                                }
                            }
                            foreach (Comment comment in deleted)
                                post.Comments.Remove(comment);
                        }
                    }
                    continue;
                }

                if (input.StartsWith("delete comments with ", 
                    StringComparison.CurrentCultureIgnoreCase))
                {
                    string text = input.Substring(21);
                    foreach (Blog blog in blogs)
                    {
                        foreach (Post post in blog.Posts)
                        {
                            List<Comment> deleted = new List<Comment>();

                            foreach (Comment comment in post.Comments)
                            {
                                if (comment.Author.Contains(text) || 
                                    comment.Text.Contains(text))
                                {
                                    blogger.Delete(new Uri(comment.EditUri));
                                    deleted.Add(comment);
                                    Console.WriteLine("{0} : {1}", blog.Name, post.Title);
                                    Console.WriteLine("-- {0} ({1})\n{2}\n", 
                                        comment.Author, comment.EditUri,
                                        comment.Text);
                                }
                            }

                            foreach (Comment comment in deleted)
                                post.Comments.Remove(comment);
                        }
                    }
                    continue;
                }

                Console.WriteLine(
                    "Can't parse. Please use 'help' to look up valid commands.");
            }


            Console.WriteLine("Done!");
        }
    }
}

Funny quote: arguing with idiots

"Never argue with an idiot. They bring you down to their level and beat you with experience."

Source unknown.

No Country For Old Men (via Reddit)

The pictures speak for themselves...

http://www.boston.com/bigpicture/2009/03/mexicos_drug_war.html

Death of journalism

So this is what passes for publishable news these days - Paul Krugman writes two pages in NYT Opinions section (http://www.nytimes.com/2009/03/23/opinion/23krugman.html?_r=1), and Reuters immediately quotes half of it on its web site: http://www.reuters.com/article/GCA-CreditCrisis/idUSTRE52M4SS20090323.

Guess which one of them was cross-linked from Reddit? You guessed right - the Reuters one.

What should have been a link to the original article, has become a copycat in its own right. A page was added to the Web. A "journalist" was paid for it. The reporting standards went down one notch further. The life continues...

The big takeover (via Reddit)

"These people need their trips to Baja, their spa treatments, their hand jobs," says an official involved in the AIG bailout, a serious look on his face, apparently not even half-kidding. "They don't function well without them."

http://www.rollingstone.com/politics/story/26793903/the_big_takeover/print

If you thought iFart and iBoobs were weird...

...behold the Cylon Detector iPhone App!

http://www.scifi.com/battlestar/iphone/

Hooray for long American tradition for consumer-driven innovation!
----
Thought about it some more, and came to conclusion that the consumers of this stuff are no weirder than Sarah Palin voters after all :-)...

On hacking cute shiny objects (via Reddit)

"Safari on the Mac is easier to exploit. The things that Windows do to make it harder (for an exploit to work), Macs don’t do. Hacking into Macs is so much easier. You don’t have to jump through hoops and deal with all the anti-exploit mitigations you’d find in Windows.

It’s more about the operating system than the (target) program. Firefox on Mac is pretty easy too. The underlying OS doesn’t have anti-exploit stuff built into it."

http://blogs.zdnet.com/security/?p=2941

This free software is very, very expensive!

New TFS documentation pushed

I blogged earlier (http://1-800-magic.blogspot.com/2009/02/everything-you-know-about-diffing-files.html) about how terrible TFS API documentation was.

This is no longer true - the latest round that just got pushed on MSDN is solid. Good job, team!

http://msdn.microsoft.com/en-us/library/bb130146.aspx

How to install TFS on a single domain-joined server

Since TFS documentation is so convoluted, I thought I would put together a SIMPLE list of what needs to be done.

Software required:

• Windows Server 2008 Standard


• SQL Server 2008 Standard Edition


• Sharepoint Server 2007 WITH SP1


• TFS 2008


• TFS 2008 SP1

All this software is available from MSDN.

Also: one domain account different from the installing user. Should NOT have admin rights. Should NOT be a local account. This is very important.

Prework:
Prepare TFS 2008 slipstreamed with SP1 as follows:

1. Copy the contents of TFS DVD AT folder to a directory, say c:\tfs\base:
       xcopy /die d:\at c:\tfs\base


2. Unpack the SP1 installer to a different directory, say c:\tfs\sp1:
       en_visual_studio_team_system_2008_team_foundation_server_service_pack_1_x86_x64wow.exe /extract:c:\tfs\sp1


3. Slip-stream the sp1:
       msiexec /a c:\tfs\base\vs_setup.msi /p TFS90sp1-KB949786.msp TARGETDIR=c:\tfs\slipstream


4. Create ISO and burn it on a DVD using ISO Recorder (http://isorecorder.alexfeinman.com/W7.htm)

Installation steps:

You must be logged as a domain user and have admin rights to the machine.

If you are installing on a VM, take snapshots between the installation steps, so if something goes wrong, you can restart easily.

1. Install Server 2008. Configure, join the domain, apply all updates. Reboot if necessary.


2. Add Web Server (IIS) role from Server Manager->Roles->Add roles. Select all components.


3. Install Windows Sharepoint Server 2007 SP1 using all defaults (non-SP1 build will not install on Server 2008).


4. Install SQL Server 2008.
   
   • Select all components to install.
   
   
   • On the page that allows you to specify the instance name, select the default instance.
   
   
   • On service accounts page, specify "Network Service" everywher (where it's editable) and make all services that can be made autostart do so.
   
   
   • Ensure Windows authentication is selected (the default), and click "Add current user" to administrators of the computer on the two pages which ask you to specify the admin user (one for SQL, one for analysis services).
   
   
   • On the page that asks you about configuring reporting, select "Install, but leave Reporting Services unconfigured" option.


5. Install TFS.
   
   • Do not change the default SQL instance name (it will be your computer name - definitely DO NOT switch it to '.').
   
   
   • Use "Network Service" where it allows you to (the default), and specify that non-admin user account from pre-requisitive section on the next page.
   
   
   • The installation will also ask you for the sharepoint admin URL. The default is incorrect. You can find the right port by opening the sharepoint admin link from the start menu.


6. RESTART THE SERVER. TFS does not prompt you to do this, but the permissions are synchronized on the service restart, so you won't be able to create a new project until you do this.

You're done.

Ensure that the client computer has Team Explorer that is upgraded to SP1 (creating project will fail in misterious ways if server has SP1 and client does not), and create a new project to test the newly installed system.

Adventures in TFS, continued

I was chronicling my attempt to install TFS on a VM at home so I have a better platform for fixing bugs related to that source control system. The beginning of this adventure is here: http://1-800-magic.blogspot.com/2009/03/adventures-in-tfs.html.

I should have given up on this thing a long, long time ago, but it started feeling like a technical challenge. One man vs. TFS. "The old man and the sea" of sorts. Although there the challenge was much less wanton :-).

Day 7. Create a new VM. This time I take a snapshot in Hyper-V every time something new gets installed.

• Windows Server 2008 + all updates


• Snapshot


• Sharepoint Server 2007 SP1


• Snapshot


• SQL Server 2008 Standard


• Snapshot

This time around TFS setup crashes after complaining that a command with half-a-page worth of command line options returned a non-zero result. Looking at winerror.h, the returned result is an unhandled exception. When I run the same command from a prompt (lots of retyping!), the exception has something to do with SQL reporting services.

At this point I give up and start reading the manual. TFS installation manual is an exercise in hyper-linking, forming a complex graph with many cycles in it. Instead of having 2-3 lists (if you are creating a single server deployment, do this, this, and this), it is an unwieldy bowl of spaghetti docs, each step cross-linking at least 5 more subtopics.

After spending 30 minutes down this maze of twisty little passages, all alike, the only difference I can find is about installing SQL server with unconfigured reporting services.

I roll back to the checkpoint after sharepoint, and this time I choose "Install but not configure reporting services".

TFS install can now complete, but I am back to the square one - trying to create the project in Visual Studio does not work - again, the error message is vague, something about reporting services.

Day 8.

I suddenly remember that I applied Visual Studio SP1 BEFORE I installed TFS client on my client machine. Everything worked so far just editing/checking files in, but I decide to reinstall SP1 just to make sure.

An hour and a half later - yes, this is how long it takes VS SP1 to install - everything works! That was the problem...

This is why we're not winning in the online world...

Adventures in TFS

Now that more and more teams are using Malevich (http://www.codeplex.com/Malevich), I am getting more and more TFS users. TFS had become quite popular at Microsoft because it integrates very nicely with Visual Studio, and of course DevDiv is using it (as they should) for dogfood.

TFS has many idiosyncrasies that Perforce does not, so occasionally I am getting bug reports for things that work nicely in Perforce/Source Depot, but not in TFS.

For a while, I was using CodePlex's Malevich enlistment for fixing bugs, but now that the bugs had become harder and harder to reproduce and require more and more setup, I decided to install my own version of TFS.

One thing that most TFS developers usually don't have to do, is TFS setup. Perforce is simple, really simple. You just copy an executable to a computer and run it - that's all. Well, you do have to open a port in the firewall, then the first time you log in, set up an admin account.

Here's what I had to go through to get TFS to work. Of course I was only working at home, at night, so a day is not really a day - it's just a few hours, sometimes even less.

Day 1. Try to install TFS on one of my servers. Discover that it does not work on 64-bit machines (huh???).

Day 2. Try to install TFS on one of my 32-bit servers. The setup at least starts now. If I use MYCOMPUTERNAME as a SQL Server target (the default instance is SQL 2008 on that box), the setup dies with many complaints about SQL Full Text Search not being installed/started/set up to start automatically. If I use '.' as an instance name, the installation goes through - up to the point where it wants a Sharepoint server. I don't have the Sharepoint server.

Day 3. I am turning on Hyper-V on one of my bigger servers. It's a dual Xeon5030 box, so 4 cores. Nice machine. Server 2008 installs almost instantaneously in Hyper-V, and the speed - I can't tell the difference. I also install SQL Server 2008.

Day 4. I have Sharepoint 2007 from MSDN, but it does not install on Server 2008. Need SP1. So I download Sharepoint 2007 with SP1. Install it. Trying to install TFS. No luck - the installation fails because it cannot find SQL Reporting Services. I am verifying - yes, reporting services are installed.

Day 5. I realize that when I use '.' it is finding a SQL 2005 instance brought in by Sharepoint. When I use MACHINENAME, it is finding SQL 2008 instance that I installed for it. SQL 2005 (that instance) does not have reporting services. SQL 2008 does not have Full Text Search as a separate service, so TFS setup is incapable to detect it. Google says I need SP1 for TFS 2008 to install on SQL Server 2008.

Day 6. Checking MSDN - MSDN does not have TFS with slip-streamed SP1. Google gets me an article that describes how to do it myself. The article is more than a bit imprecise, so I am reproducing the actual steps here:

• Copy the contents of TFS DVD AT folder to a directory, say c:\tfs\base (xcopy /die d:\at c:\tfs\base).


• Unpack the SP1 installer to a different directory, say c:\tfs\sp1 ( en_visual_studio_team_system_2008_team_foundation_server_service_pack_1_x86_x64wow.exe /extract:c:\tfs\sp1). Why does this have x64wow in its name despite the fact that it won't work on x64, wow or no wow - no idea.


• Slip-stream the sp1 (msiexec /a c:\tfs\base\vs_setup.msi /p TFS90sp1-KB949786.msp TARGETDIR=c:\tfs\slipstream)

You're done, the slipstream directory contains TFS 2008 SP1. The rest of the stuff on the TFS DVD (build, TE) is not slip-streamable, so just the TFS part is what you need. Make an image from this directory and voila - this will actually install on Server 2008 with SQL Server 2008!

I am trying to now create a new TFS project. I can connect to the server, but when I do try to create a project, it fails.

TF30004: The New Team Project Wizard encountered an unexpected error while initializing the Microsoft.ProjectCreationWizard.Reporting plug-in.  

Explanation 
TF30171: The Microsoft.ProjectCreationWizard.Reporting plug-in used to create the new team project could not be initialized and returned the following error: TF30224: Failed to retrieve projects from the report server. Please check that the SQL Server Reporting Services Web and Windows services are running and you have sufficient privileges for creating a project..  

User Action 
Contact your Team Foundation Server administrator.  

I am trying to investigate this. It turns out that I have no access to SQL Reporting services - I've installed SQL server as a local admin, and despite the fact that I've added Domain Admins, my user name, and everything to the admins, it did not work.

I decide that the easiest thing would be to just uninstall everything, and reinstall it from a normal user account.

Unfortunately, I start uninstallation from SQL Server. After I uninstall SQL Server, TFS installer crashes on uninstall. It looks like the VM is toast...

To be continued...

A very wonderful opportunity...

...has just presented itself in the form of you potentially paying $1000 for 4 gigs of RAM:

http://store.apple.com/us/configure/MB419LL/A?mco=NDE4Mzg5MA

Also, did you know that FileMaker is "#1 name in databases"? You do now!

Just how dumb are we, really?

http://www.gallup.com/poll/116065/Americans-Views-Bank-Takeovers-Appear-Fluid.aspx

Gallup poll asks half of their polling sample whether they are for or against temporary "takeover" of the banks, and the other half whether they are for or against temporary "nationalization". The rest of the question is identical.

On the takover question: 54% for, 44% against, 3% no opinion.

On the nationalization question: 37% for, 57% against, 6% no opinion.

These people are allowed to vote?

Leading scientists still reject God

"You clearly can be a scientist and have religious beliefs. But I don't think you can be a real scientist in the deepest sense of the word because they are such alien categories of knowledge."

http://www.stephenjaygould.org/ctrl/news/file002.html

Eventually we must stop the stupid blabber about how religion and science are compatible, and just admit that they are not.