Corn-fed venison

Funny as hell...

http://www.bloggingwv.com/corn-fed-venison-it-looked-good-on-paper/

Religious discrimination...

PART IV. CRIMES, PUNISHMENTS AND PROCEEDINGS IN CRIMINAL CASES

TITLE I. CRIMES AND PUNISHMENTS CHAPTER 272. CRIMES AGAINST CHASTITY, MORALITY, DECENCY AND GOOD ORDER Chapter 272: Section 36. Blasphemy Section 36. Whoever wilfully blasphemes the holy name of God by denying, cursing or contumeliously reproaching God, his creation, government or final judging of the world, or by cursing or contumeliously reproaching Jesus Christ or the Holy Ghost, or by cursing or contumeliously reproaching or exposing to contempt and ridicule, the holy word of God contained in the holy scriptures shall be punished by imprisonment in jail for not more than one year or by a fine of not more than three hundred dollars, and may also be bound to good behavior.

http://www.mass.gov/legis/laws/mgl/272-36.htm


Apparently, atheists are prohibited from holding office in...

• Arkansas (Constitution Of The State Of Arkansas Of 1874. Article 19. Miscellaneous Provisions. § 1. Atheists disqualified from holding office or testifying as witness. No person who denies the being of a God shall hold any office in the civil departments of this State, nor be competent to testify as a witness in any Court.)


• Maryland (Article 37 of the Declaration of Rights of the Maryland Constitution That no religious test ought ever to be required as a qualification for any office of profit or trust in this State, other than a declaration of belief in the existence of God; nor shall the Legislature prescribe any other oath of office than the oath prescribed by this Constitution.)


• North Carolina (North Carolina State Constitution, Article VI, Section 8: Sec. 8. Disqualifications for office. The following persons shall be disqualified for office: First, any person who shall deny the being of Almighty God.)


• South Carolina (South Carolina State Constitution, Article VI, Section 2: No person who denies the existence of the Supreme Being shall hold any office under this Constitution.)


• Tennessee (The Tennessee Constitution, Article IX, Section 2 No person who denies the being of God, or a future state of rewards and punishments, shall hold any office in the civil department of this state.)


• ...and Texas (The Texas Constitution, Article I, Section 4: No religious test shall ever be required as a qualification to any office, or public trust, in this State; nor shall any one be excluded from holding office on account of his religious sentiments, provided he acknowledge the existence of a Supreme Being.)

http://www.freethoughtpedia.com/wiki/Anti-atheist_laws

The most bizzare part is not that there are these laws - they mostly date to 1800s. It's that they have not been challenged yet!..

Update: this article from Rolling Stones should be good for a decade in jail in Massachusetts... http://www.rollingstone.com/politics/story/20278737/jesus_made_me_puke/print...

Panasonic HDC HS9 product review

We've got a new 1080p camcorder yesterday - Panasonic HDC HS9, just in time for the summer vacation season.

This thing has got to be a technological marvel - it captures and encodes video in 1080p format - it hasn't been more than 2 years when your typical computer could not even PLAY in this resolution, let alone record. This records full resolution on the fly, and stores the results on a 60GB hard drive.

Good things first.

The specs are great for the price - 1080p recording, 3 CCDs, 60GB hard drive. It is also small and light - the size of a palm. It produces very decent videos outdoors, in full daylight (but see below).

The sound it records is excellent - really. 5.1 surround sound, and it really truly sounds like a good 5.1 surround recording.

And it is relatively inexpensive at ~$750.

The best thing about this camera is the convenient output - it basically produces BluRay disk file structure. You can copy the directory from the camcorder and drop it on the PowerDVD HD window, and it just plays. The interface is USB, it appears as an external hard drive, so it's drag and drop to save the files, and then drag and drop to play them. Very nice.

Now, the bad things. In general, the design of this thing is terrible - across the board. I am sorry to say this, but the UI designers for it were idiots - there's no other word to describe this.

For starters, the USB connector is behind the LCD screen. So to connect it to a PC one has to open the screen, peel away a little plastic cover, and connect the USB cable. And while it is connected, the LCD screen stays open. If you drag the cable and it falls on the floor, the screen is almost guaranteed to break off.

This is not all. If you have it connected to a PC, you MUST have it on external power - it cannot use USB, OR its own battery. Now the power connector is behind the battery - to connect it, you have to take the battery away.



So when you're copying the files, you have two cables plugged into the thing, the little plastic USB slot cover hanging on its plastic strip, the LCD screen is open, and the battery is lying to the side. What a mess.

This of course also means that the you can't charge the battery while it's inside the camcorder. You have to take it out and insert it into the charger. You might hope that you can charge it while copying the images from the camera, but no. The charger does not charge the battery if the camera is plugged in. How silly is this?

It actually does matter. The thing produces ~1.3GB every 10 minutes, and the battery resource is ~70 minutes, after which it takes 1.5 hours to charge. Also, 1.3GB video takes roughly 2.5 minutes to copy off to the PC, so for 70 minutes worth it is another 20 minutes. If they were to allow charging while camcorder is connected, it would mean 20% savings in time it takes to get the camcorder ready to shoot again.

Now, there IS space on the case where the connectors - power and USB - might have belonged more logically. This space is used by an enormous SD card slot. According to the manual, the purpose of SD card slot is to shoot where the hard disk cannot be used - at the elevations of above 3000 meters (for Seattleites, it's just below Camp Muir on Mount Rainier), or in high-vibration environments such as a dance club.

I would much rather have SD slot hidden, and the USB connector exposed, on the expectation that most people would be using the hard disk most of the time, and even when they do use the SD card they would still use USB to transfer the data.

Speaking of the manuals... real men don't use them, do they? Well, I'd love to see a real man using THIS camcorder :-). Without reading the manual, this thing is utterly useless. Looking at the menu system, you cannot ever guess what is where. The time settings for example are spread across two menus - setting the time and the time zone (which is not called time zone by the way) is in "Basic" menu, setting the time format is in "Setup".

I still did not find where one changes video options such as low light settings - the camcorder offers it as a prompt when it detects the low light situations, and this is how I select it, but I have no idea how to find it in the menus. Neither I know how to turn the light on for the low-light setting. I didn't get to it in the manual yet.

These are the glaring UI problems - there are plenty of minor nits. For example, the charger shows a green LED when it charges, and it goes out when the charging is done. Most consumer electronics things that I owned either blink the LED when it charges, or show the yellow LED that goes green when the charging is done.

The UI problems transcend the device itself.

The software tries to use skins, except the message boxes that it shows are not skinned. They are just plain stupid Windows message boxes and look out of place in the UI that otherwise tries to look like a media player. And they are everywhere - any action brings up a message box (you click on "transfer video" button - the message box opens saying "Transfer video?" (Yes/No)). Kinda like Vista, only worse.

The default location where software copies the data - on XP at least - is "All Users\My Documents\My Pictures". Good luck finding this in YOUR "My Documents" folder after this. Both videos and pictures go there, in a strange structure - the videos for example are in the PRIVATE folder. How DID they know what it was I was shooting?!

There are a few seemingly arbitrary limitations in the software - it plays media from the SD card (while it is in the unit), but not from the unit's hard disk, for example. Instead it for some reason does support playing it from a DVD. Why bother with this piece of functionality?

The error paths in the software are not tested. This is what it produces when you try to install it on Media Center.


A worse problem than the terrible UI is its performance in the low-light situations. As I wrote it records excellent videos in daylight. But in the evening the videos become grainy and blurry at the same time, as if it had problems with auto-focus. Here is a frame from a video shot at daylight:

And here's one taken during the evening (yes, I was using the low light mode):

I must admit though I do not have a frame of reference on this though - I don't know how other camcorders perform in similar situations. The review on Amazon says that Sony produces equally terrible images in low light conditions.

So net/net I would give it 3.5 stars out 5 - just barely enough to not contemplate returning it back where I got it from.

Are you on the list?

"Senior government officials have leaked detailed information about a database of 8 million Americans targeted for detention in case of a declared national emergency."

http://www.dailykos.com/storyonly/2008/5/20/21950/7576/933/518756

Premature optimization is the root of all evil

Every once in a while I come across a code review where there is a small inefficiency in the code which can be easily corrected, but where an author invokes the ghost of "premature optimization" to justify keeping it this way.

Today's example (Java)...

Map map;
...
for ( ; ; ) {
  ...
  if (!map.containsKey(key))
    continue;
  Object x = map.get(key);
  ...
}

This does the hash lookup twice - first when checking whether it contains the key, and second when retrieving the value. In the case where objects stored in the map are never null, the time spent in this code can be cut in two by just doing this:

Map map;
...
for ( ; ; ) {
  ...
  Object x = map.get(key);
  if (!x)
    continue;
  ...
}

Is this a premature optimization?

What about this (C++):

bool process(string &in, string *out) {
  *out = in;
  string x, y;
  if (!subprocess(in, &x, &y))
    return false;
  *out = x + '/' + y;
  return true;
}

"*out = in;" being something on the order of several hundred of instructions (http://1-800-magic.blogspot.com/2008/04/stl-strings.html), the same code can be rewritten, without the loss of readability, as follows:

bool process(string &in, string *out) {
  string x, y;
  if (!subprocess(in, &x, &y)) {
    *out = in;
    return false;
  }
  *out = x + '/' + y;
  return true;
}

Would this be a premature optimization as well?

The term premature optimization, originally applied by Knuth and Hoare to making design trade-offs to optimize to clock-level efficiency is most often misapplied to mean that how you write the code does not matter - we'll figure out what's slow and optimize it later.

There are a few problems with this approach. First, as Hoare writes himself...

"I've always thought this quote has all too often led software designers into serious mistakes because it has been applied to a different problem domain to what was intended. The full version of the quote is "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." and I agree with this. Its usually not worth spending a lot of time micro-optimizing code before its obvious where the performance bottlenecks are. But, conversely, when designing software at a system level, performance issues should always be considered from the beginning. A good software developer will do this automatically, having developed a feel for where performance issues will cause problems. An inexperienced developer will not bother, misguidedly believing that a bit of fine tuning at a later stage will fix any problems."

(Emphasis mine.)

There are certain systemic decisions that a designer makes before writing the code (do I use STL? Do I use Java?) that are extremely difficult to undo once the code is complete - the performance/memory/code size implications may end up spread across the entire system - perhaps across multiple layers of the software stack - and are extremely hard to localize and "fix".

What's worse, the "fix" usually is an even worse hack.

Consider a developer who applies Java to the wrong application domain (http://1-800-magic.blogspot.com/2007/11/domain-languages.html) - e. g. media processing - and ends up with a very slow system.

This developer might be compelled to rewrite parts of it in C++ and sprinkle the native code in various parts of the otherwise managed system where he or she has found one could gain the most performance improvement by profiling.

Of course this makes the design more complicated and the code less readable - now we have some parts of the system implemented in one language, and other seemingly random parts - in another. This flies right in the face of the originally stated goal of "cleanliness" of the design. It also makes debugging and correctness verification tasks much harder.

Second point I want to make is the implementation cleanliness. While one might argue that checking whether the hash contains an element or not more clearly expresses the intent of the developer, it is actually in the eyes of the beholder.

When I look at the original Java code snippet, the first thing that comes to my mind is - "oh, (s)he must be storing nulls as possible key values, so the check needs to be done upfront. But wait, (s)he then dereferences the value without checking for null? A bug, perhaps?.."

And when I look at the C++ snippet, I think that returning "false" rather than "true" is perhaps the common case, since the code is clearly optimized for it (it does more work than necessary in the "true" case, but that probably does not matter because we never follow that code path in reality).

So as you can see, the opinions on what's readable - explicit check/upfront initialization vs. checking for null/error case initialization - might diverge. For me doing the extra work makes code less readable - I expect that this was done for a reason, and would start searching for this reason. Another developer might like the more explicit but redundant code better.

Since we don't know who would be reading our code, and what his or her preferences might be, we should not trade ephemeral aesthetics vs. the very real watt-hours burned by the CPU executing suboptimal software.

The third reason for chosing the more efficient programming style is because it will make one a better developer. To reiterate the Hoare quote, albeit a bit out of context, "A good software developer will do this automatically, having developed a feel for where performance issues will cause problems."

The code above is worth noticing and correcting simply to make sure that the developer trains himself or herself to avoid writing the same code in the future (and perhaps, this time, in the critical path). After a correction or two, the engineer will start producing efficient implementations automatically.

I consider this point of style that separates the men from boys (and the women from girls, to be politically correct). If you look at code written by great developers, it is efficient without sacrificing readability, non-redundant without being obscure. It implements the algorithm - even in a pseudocode - in a way that makes it completely clear what author wants to do but does not sacrifice one clock of CPU time.

Take a look at the code in CLR (http://www.amazon.com/Introduction-Algorithms-Thomas-H-Cormen/dp/0262032937), Sedgewick (http://www.amazon.com/Bundle-Algorithms-C%2B%2B-Parts-1-5/dp/020172684X), or David Hanson's "C Interfaces and Implementations" (http://www.amazon.com/Interfaces-Implementations-Techniques-Addison-Wesley-Professional/dp/0201498413). Even when written in pseudocode, it is an amazingly effective pseudocode. It manages to achieve this effectiveness without sacrificing the readability.

Learn to do this, and you're going to be a great developer. Take up the religion where writing effective code by default is a "premature optimization" - and consign yourself to a career, as Joel puts it, of copying and pasting a whole bunch of Java code :-).

More on premature optimization here: http://www.acm.org/ubiquity/views/v7i24_fallacy.html.

Democracy and deference

Mark Slouka traces a very interesting parallel between strongly hierarchical models of most business organizations, and what it means for the domestic politics.

The central thesis is that we're so used to unquestioningly follow orders at work, that we translate the same paradigm into politics, and it becomes just too easy to forget that a president is not a sovereign ruler, but rather a servant of the people. And the president is only too happy to behave like a king since the the population allows it.

"Turn on the TV to almost any program with an office in it, and you'll find a depressingly accurate representation of the "boss culture," a culture based on an a priori notion of-a devout belief in-inequality. The boss will scowl or humiliate you... because he can, because he's the boss. And you'll keep your mouth shut and look contrite, even if you've done nothing wrong... because, well, because he's the boss. Because he's above you. Because he makes more money than you. Because - admit it - he's more than you.

This is the paradigm - the relational model that shapes so much of our public life."

http://www.harpers.org/archive/2008/06/0082039

I think there's a lot of truth to this. I was once in the path of a senator at the Consumer Electronics Show in Las Vegas. His bodyguards were literally shoving people out of his way as his excellency was moving across the exhibition floor.

I could never imagine this was possible - even in Soviet Russia I did not ever see party officials (or rather, their guards) behave like that.

Whether this phenomenon is caused by the corporate culture there's no way of telling for sure, but I'd say it is probably not a bad guess. I've seen people change almost completely when they changed their managers, to an extent that their behavior was difficult to recognize.

Einstein's letter on religion goes for 170000 pounds

The guide price was 6000-8000 pounds. I was bidding on it, too, via an absentee bid. My puny offer of 9000 pounds was not even close :-(.

This was the only known written communication where Einstein expresses his views on religion directly.

My wife's reaction, which I do share was - "hope they didn't buy it to destroy".

"... The word God is for me nothing more than the expression and product of human weaknesses, the Bible a collection of honourable, but still primitive legends which are nevertheless pretty childish. No interpretation no matter how subtle can (for me) change this. These subtilised interpretations are highly manifold according to their nature and have almost nothing to do with the original text. For me the Jewish religion like all other religions is an incarnation of the most childish superstitions. And the Jewish people to whom I gladly belong and with whose mentality I have a deep affinity have no different quality for me than all other people. As far as my experience goes, they are also no better than other human groups, although they are protected from the worst cancers by a lack of power. Otherwise I cannot see anything 'chosen' about them.

In general I find it painful that you claim a privileged position and try to defend it by two walls of pride, an external one as a man and an internal one as a Jew. As a man you claim, so to speak, a dispensation from causality otherwise accepted, as a Jew the priviliege of monotheism. But a limited causality is no longer a causality at all, as our wonderful Spinoza recognized with all incision, probably as the first one. And the animistic interpretations of the religions of nature are in principle not annulled by monopolisation. With such walls we can only attain a certain self-deception, but our moral efforts are not furthered by them. On the contrary.

Now that I have quite openly stated our differences in intellectual convictions it is still clear to me that we are quite close to each other in essential things, ie in our evalutations of human behaviour. What separates us are only intellectual 'props' and `rationalisation' in Freud's language. Therefore I think that we would understand each other quite well if we talked about concrete things.

With friendly thanks and best wishes

Yours, A. Einstein."

http://www.relativitybook.com/resources/Einstein_religion.html

Price/performance

A picture takes the space of ten thousand words, but is worth only a thousand.

(Based on a typical low-res jpeg size of ~60k, 5-6 letters per word, and a famous saying :-).)

Evil web advertising...

My computer was quite sluggish today. At first, I blamed it on Gmail - I am on dev cluster, which gets a new version daily, without any testing, so I hit various bugs in it - including the performance bugs - very often. But after I closed Gmail the sluggishness persisted, so I decided to investigate.

Take a look at the following two pictures. This one has CPU utilization at 10%.


And in this one, the CPU utilization has dropped to zero.


The only difference - the little "free smileys" advertizing craplet that has crept into the background. I had a few of them at some point, and the CPU was at constant 50%...

Does you XP box reboot incessantly after SP3 was installed?

The problem and the solution is here:
http://msinfluentials.com/blogs/jesper/archive/2008/05/08/does-your-amd-based-computer-boot-after-installing-xp-sp3.aspx

A bunch of my Media Center computers are AMD-based cheap motherboard+CPU combos from Fry's. I look forward to having lots of fun fighting this soon... not!

Code Reviews with a Smile

Google Seattle has a weekly event called "Gathering" - a team meeting for all engineers in the office where anyone can talk about anything of interest to Googlers. Topics range from overviews of various projects people are working on, to discussions of the new technologies, to presentations by various researchers from academia.

I took the last couple of slots to talk about the human aspects of the code reviews.

I was doing a lot of these lately. Being a readability reviewer for JavaScript guarantees you at least one code review per week, typically consisting of 3 or 4 iterations. Plus, I have a bunch of readabilities in other languages, and often volunteer to review code for teams who have nobody with readability (more on readability at Google here: http://1-800-magic.blogspot.com/2008/01/after-8-months-no-longer-noogler.html).

The talk turned out a success, and so I decided to blogify it :-).

When I was at Microsoft, I was no fan of mandatory code reviews. I thought it was impractical to have every line of code read by someone else, and I also was afraid that people would use code reviews as a substitute for testing (to an extent, this does happen at Google, which has much smaller test organization than Microsoft).

Well, Google does require code reviews for every check-in, so I had to convert - and I did. As it often happens, new coverts become zealots :-). So not only I ended up enthusiastically supporting the system, but joined Google Readability team as well.

Why did I convert? Basically, I looked at the code base, and was impressed. Microsoft does not have a unified style, and a lot of the code looks jagging to the eyes of a person who did not write it.

It's like an accent - small groups of people that communicate in relative separation from the main body of the language carriers develop it. In case of developers whose only communication is the compiler, a very strong "personal" accent develops, and the resulting code takes an effort to understand by an outsider.

The code reviews smooth the accent by expanding the group of people who speak the same code. Having a single style guide expands the accent to the entire company, so any part of the code looks like you wrote it yesterday. For me at least it improves the productivity dramatically.

Also, I noticed that the developers tend to write better code upfront if they expect that someone else will be looking at it - out of the sheer embarrassment :-).

But beyond the company benefits, there are two reasons why I personally love doing the code reviews.

First is because I learn a lot from them myself. Being exposed to code written by people in different corners of the company is the best way to keep abreast of the many projects you would otherwise have never heard about. Right now I am in the process of reviewing code that I am pretty sure I will use myself - and if not for the code review, I would never knew it existed.

Second is because it gives me an opportunity to teach. After more than 20 years of programming computers, there's a bunch of stuff I know, and recalling it during code reviews keeps it alive :-).

However, once we take the code out of the purely machine environment of compile-run-test-check in, it becomes a human communication. Human communications have the emotional aspect which is missing entirely from the computer-bound interaction of reviewless programming.

Thus code reviews add an entirely new dimension - above and beyond technical aspects, they are an instance of a human communication, and a very sensitive one at that - because in the process of a code review, one developer render opinion about the work of another engineer. Tread carefully!

Studies upon studies have shown that people use the emotions first, and the logic next. So the emotional rapport between a reviewer and the reviewee can have a bigger effect than the information that trades hands in the process. "How" can - and often does - become more important than "what".

I think about most human interactions as the bank transactions involving trust. You make a deposit into your account when you do something that the other person likes. You make a withdrawal when you have the other person do something that you like (but the other person might not).

In terms of code reviews, what is important for a reviewee? How could a reviewer increase his or her balance?

I think that the most important things for a reviewee is the latency - the sooner the review (or at least an iteration of the review) is done, the better.

While a code review is outstanding, a reviewee is often barred from continuing the work with the same set of files, because the newer changes could involve the same code that is not yet checked in pending the review.

If the latency is low, even big changes requested by the reviewer are easier to make - but if the reviewer sits on the code for two weeks, then requests a total rewrite, and then sits on it for two weeks again before declaring that the first version was better after all - it's entirely a different matter. The tension builds.

Another case where the stress tends to build up if when the reviewee works against a hard deadline. But the reviewer often does not live on the same shipping cycle, and is not taking the rush into account. This is good, of course - it helps ensure that the bad code does not get checked in purely to satisfy the timing constraints. But it is very important that the reviewer realizes that the pressure is building - and acts in ways that help relieve it.

What else is important for a reviewee? I'd say the style with which the reviewer acts, the civility of the communication. People generally do not tell their office mate - "Hey, you! Fetch me a chair!". But I've had many, many times seen the review comments say "Rename this variable to foo."

If you and your reviewer has worked together for a while, and built up considerable deposit in their trust account, it is OK to spend a little bit of it to conserve typing. But do not save on civility when reviewing the code for a stranger!

Here are my recommendations for the reviewer:

• Respect the reviewee's time
  
  
  
  • Maintain quick turnaround
  
  
  • Do not ask for small tweaks that do not matter. Does this variable really need renaming? If it were your code, would you care to check out, change, and retest 10 files just to rename this function or not?


• Order not, suggest
  
  
  
  • “Consider naming this foo because this is what it’s named everywhere else in the codebase.”
  
  
  • “If you use bar instead, it could save you some code.”
  
  
  • “Moving baz here could shave a few clocks from the execution path”


• Respect reviewee's opinion
  
  
  
  • (S)he has spent days thinking about this. You have spent less than an hour...
  
  
  • In case of conflicting approaches (AKA religious disagreements), it’s the reviewee who ultimately owns the code – (s)he should have the priority


• Praise! Nothing else creates good will more effectively than this.
  
  
  
  • "Great CL, thank you for doing this!"


• Golden rule: review others’ code as you want your code to be reviewed

Now, let's look at it from the reviewer perspective. What does one want when he or she reviews someone else's code?

I personally like my efforts to be recognized. There is a reason I am doing this, right? I want my opinion to be respected :-).

As a reviewer, I want my time to be used effectively.

And, it goes without saying, I want the product's code base to be as good as possible.

What does this mean for a reviewee?

• If it’s not too hard, it is often easier to just do it
  
  
  
  • If you find yourself pushing back on almost everything your reviewer suggest, one of you may be unreasonable
  
  
  • If you do it often with different people, the unreasonable person is you…


• Respect reviewer's time
  
  
  
  • Smaller change lists
  
  
  • More comments, both in code and in change description


• Recognize stellar reviewers at the performance review time
  
  
  
  • A note dropped to the person's manager endorsing a stellar reviewer will make his or her day


• And, of course, the Golden rule again!

Have your own story of a code review that went badly because of lack of rapport between the people involved? Did I miss an important point? Write about it here!

Vista: how to open the box

Windows Help and How-to page on opening the box Vista is shipping in:

http://windowshelp.microsoft.com/Windows/en-US/help/2e680b8d-211e-41c5-a0bf-9ccc6d7e62a21033.mspx%5C

A friend had sent this to me. I must confess, I fumbled with the box, too, for far longer that I should have. So the post link is in order.

But... who designed this thing?! And I mean... the whole thing :-(...

The Nazis: A Warning from History

When Ken Burns' famous "The War" came out on DVD, we rented it just in time for the Spring Break so our daughters could watch it with us. But after doing the first two disks, we were too bored to continue.

I found "The War" to be very repetitive and shallow, and it suffered very much from the US-centric view of history, skipping almost entirely over anything that was going on in Europe until the landing of American troops in Italy (and then it focused on, well, you guessed right - American troops in Italy!). Which means that it missed about 70% of the conflict (http://1-800-magic.blogspot.com/2008/05/how-us-has-won-world-war-ii.html).

Being overhyped to the high heavens by the media did not help of course - it had set my expectations high, and the movie came way short...

"The Nazis: A Warning from History" which we rented a week ago, was only two disks to Burns' six, but I learned more from the first 15 minutes of it than from watching the first two DVDs of "The War".

The film covers the period from Weimar Republic to right before the fall of Berlin.

Like the War, it focuses on interviewing the eyewitnesses - but the people who participated in the European theater where majority of the action was happening.

Most of the interviewees were former Nazis, former soldiers, leaders of small Hitler-Jugend, just German citizens at the time. I think the focus of the movie was the banality of evil - that Hitler and his senior henchmen aside, the war would have been impossible without willing and sometimes enthusiastic cooperation of the "simple folks", although it does give a good insight on how the Nazi government operated on all levels.

In several cases, the movie research teams had unearthed documents that the interviewees probably would preffer to have never had existed - a letter denoucing the neighbour, court documents depicting someone's participation in the death squads, etc, and confronted the interviewees with them. Responses varied, but to fully appreciate the situation, you have to see the movie.

Highly recommended!

Shareholders Revolt Against Bloated CEO Pay

http://www.alternet.org/workplace/84468/

YHOO offer withdrawn, but the stock price has not recovered

So, the spell of insanity must have passed over, and Microsoft has withdrawn Yahoo offer. This is a good thing for Microsoft - the combination would have never worked: http://1-800-magic.blogspot.com/2008/02/50b-down-drain-or-microsoft-bids-for.html.

The bad news, however, is that destruction of the shareholder value is not as easily reverted.

Look at today's stock quote. When the proposed deal was announced ('J' on the chart below), Microsoft has dropped more than $2 per share. Now that the deal was withdrawn, it has rebounded - but by a mere 40 cents...

IT security as an impediment to developer's productivity

My first Computer Science teacher liked to tell this story. In the late 1940s there were 3 different classified projects to design the first computing platform in Russia. All three were run by the military, and as is typical for military designers in Soviet Union, all three were working in complete isolation from the rest of the world and each other. Paranoid times, you see: Stalin was imagining the new types of enemies of the state every day.

So one of these projects was falling behind quite a bit, and the leadership decided that it was hopeless, and declassified it. So the team could now go to the conferences, talk to other people, and engage in normal life of a research project, including a lot of information sharing.

This project suddenly had a turn-around, and produced the BESM line (http://en.wikipedia.org/wiki/BESM), which became the workhorse of Soviet computing for the next 40 years, sort of Warsaw Block IBM-360. The first version came out in 1952, the production of the last one stopped in 1987.

The other two projects stagnated and were eventually killed.

Moral of the story: secrecy is antithetical to research.

When I started working at Microsoft in 1998, I was startled to realize that the campus was not connected to the Internet at all. I had to go to the company library which had a few workstations on tap to buy tickets from Expedia. This was of course in the name of security, least one can steal the precious Windows source code.

Surprise, surprise, Microsoft struggled to win against Netscape, despite the fact that the company had far more resources to pour into the browser wars, had great programmers, and expertise in shipping.

Throughout the years I worked at Microsoft, security concerns of corporate IT were always in the way of me doing the work I was hired to do (and passionately wanted to do).

First, the IT always messed with the VPN access to the corporate network. It started just like any other VPN - you click on the icon, enter your password, and in a few seconds you are connected. This was not "secure enough", so they added a step that checks if the computer originating the VPN session has critical updates installed. Then they expanded on this brilliant thought by adding more and more checks. Then they started to require smart cards for access.

As a result, towards the end of my tenure, I had to wait at least a minute to connect when I wanted to work from home, usually more like a minute and a half.

It is kinda obvious that a company should really appreciate if people want to work for it more than standard business hours, and should make doing so as easy as possible. Google gets it by the way - connecting to work is easy (under 5 seconds in most cases), everyone gets a laptop, and they buy you a big monitor for work use at home. Microsoft doesn't.

Today, Microsoft have all but removed the VPN access to its corporate network, and replaced it with remote desktop proxy that allows people to connect directly to PCs at work through the RDP sessions. I always laugh at my wife as I observe her doing it - there are 3 dialog boxes where she has to enter her password and various PINs, and the process takes minutes...

And of course if one cared, it would still be easy to write a virus that would penetrate RDP connection if the client is infected - all it needs to do is detect when the desktop is idle for a long time (so the user must have gone away), send a keystroke emulating Ctrl press every 5 minutes so the server desktop does not lock, and then inject keystrokes into RDP client queue to have the server run something off the internet. Mission accomplished!..

Inside Microsoft, the IT security interferes with the developer's productivity as well. If you are an office worker, you probably don't notice it, because your only computer is fully managed by IT, and there's not much you're doing with it anyway.

But if you're a developer, have multiple test boxes (or devices), then the security is in the way big time. Your off-domain test hardware can't connect to anything. You can't debug it in a lot of cases, and you have to jump through a lot of hoops before you do when you can.

A guy who worked for me in my previous job went as far as removing his computer from Microsoft's domain - so much IT security was interfering with his abilities to get stuff done. He read email through the web interface.

Nobody has a way to really measure it, but my gut feeling is that Microsoft loses probably 5-10% of the productivity to the security monster. If you estimate that there are probably ~20000 people in Microsoft's test, dev, and PM orgs, that would be between a thousand and two thousand people. A size of a whole division.

And I am fairly sure Microsoft is not even close to being the worst company as far as IT impact on developers goes. I've heard about companies where developers don't have admin rights to their machines, so they can't install any software beyond that installed by IT. How scary is this - the developers being trusted with the future of their company, but not with their own computers...

The big problem with IT security is that people who make decisions of how to implement it in an enterprise are usually not engineers themselves, do not really understand the risks. And they are not the business people, either, so they do not understand the tradeoffs. They are hired to prevent, and they do prevent. And the best way to prevent code from being leaked, is to make sure that no code is written, so there's nothing to leak :-).

How US has won the World War II

The undisputed common knowledge in the US is that IT has won World War II, more or less by itself. The other countries are really mentioned, except as victims.

I keep reading stuff like this over and over again: "We have to do to them what the Americans did to the Nazis. Kill all their leaders. Kill all the collaborators. Then, we'll find those willing to make peace." http://www.macleans.ca/world/global/article.jsp?content=20080423_11237_11237&page=3 (This particular quote comes from US's 51st state - Israel, but it perfectly mirrors popular opinion in the "mainland").

Here are some numbers though (from http://en.wikipedia.org/wiki/World_War_II_casualties).

Country       Military casulaties   Total casualties
Soviet Union           10,700,000         23,100,000
Germany                 5,533,000          7,293,000
United States             416,800            418,500

80% of German military deaths were on the Eastern Front.

In military deaths, US is behind Yugoslavia, Japan, China, Germany, and Soviet Union, and just barely above UK.

In total deaths, US is behind United Kingdom, Italy, France, Hungary, Romania, French Indo-China, Yugoslavia, India, Japan, Indonesia, Poland, Germany, China, and Soviet Union, and just above Lithuania and Czechoslovakia.

So in all actuality, US involvement in WWII was far, far, far, less than most countries in Europe. And US contribution to winning the war was closer to that of France and UK, and far, far, far, far behind Soviet Union, where Germans lost 80% of their army and where it was broken in Stalingrad (http://en.wikipedia.org/wiki/Battle_of_Stalingrad) and Kursk (http://en.wikipedia.org/wiki/Battle_of_Kursk) a full year before Allies landed in Normandy.

Now, obviously, self-aggrandising is not an American phenomenon. Every country practices a healthy dose of it.

What is super dangerous in this reading of history ("U.S. has won World War II") is that it creates an idea among American population that wars are cheap, and easy to win. Look, we won in the worst war ever to hit human civilization, and most people barely noticed. (Yes, as a percentage of population, US lost... 0.32%. That's one out of three hundred. As compared to Soviet Union's 13.71% - one out of six.)

Hence, the Rambo mentality.

Hence, Iraq.

But it could have been worse, much worse.

In mid-80s there were two movies that came out in Soviet Union and United States, both dealing with the world after the nuclear catastrophe.

The Russian movie was "Dead Man's Letters" (http://en.wikipedia.org/wiki/Dead_Man's_Letters), and depicted the world that was dead.

The American movie was "The Day After" (http://en.wikipedia.org/wiki/The_Day_After), and had shown farmers removing the thin layer of soil that was irradiated, and preparing for the new crop. The message was - the life continues. We can win.

The Soviet Union is now history (look, America has won again! Just like it did with the Nazies!) but the myth of the military power on the cheap lives on - Hillary is now ready to obliterate Iran (soon to be nuclear power), and compared to the morons that are running the show now, she's the sane one.

It's funny, but the reality with Iran is probably going to turn out quite differently - by switching oil trading from dollar to euro it is they who are more likely to obliterate the US, not the other way around.

Meanwhile, the military hardware is being built. No health insurance though, we can't afford that...

5/5/2008:
In many countries such nationalism arises from a pent-up frustration over having to accept an entirely Western, or American, narrative of world history—one in which they are miscast or remain bit players. Russians have long chafed over the manner in which Western countries remember World War II. The American narrative is one in which the United States and Britain heroically defeat the forces of fascism. The Normandy landings are the climactic highpoint of the war—the beginning of the end. The Russians point out, however, that in fact the entire Western front was a sideshow. Three quarters of all German forces were engaged on the Eastern front fighting Russian troops, and Germany suffered 70 percent of its casualties there. The Eastern front involved more land combat than all other theaters of World War II put together.

http://www.newsweek.com/id/135380/page/5

LHC! LHC! LHC! LHC! LHC!..

http://www.ted.com/index.php/talks/view/id/253

Funny anti-ad for Mac Air

A product I would be proud to have designed...

http://www.intuitivesurgical.com/products/da_vinci_video_overview.aspx

Warren Buffet says recession is going to be bad, invests in chewing gum!

"NEW YORK (Reuters) - Warren Buffett, the world's richest person, said on Monday the U.S. economy is in a recession that will be more severe than most people expect.

Buffett made his comments on CNBC television after his Berkshire Hathaway Inc (BRKa.N) (BRKb.N) agreed to invest $6.5 billion in the takeover of chewing gum maker Wm Wrigley Jr Co (WWY.N) by Mars Inc in a $23 billion transaction.

"This is not a field of specialty for me, but my general feeling is that the recession will be longer and deeper than most people think," Buffett said. "This will not be short and shallow.

"I think consumers are feeling gas and food prices," he added, "and not feeling they've got a lot of money for other things.""

http://news.yahoo.com/s/nm/20080428/bs_nm/buffett_recession_dc_1

Chewing gum prices are positively correlated with the prices of food - the more expensive the food is, the more chewing gum is used to replace it :-)...

The next development - George Soros shorting the weight-loss industry - to be announced later!

Computer Science lectures

This blog has a bunch of pointers to what looks like a nice collection of CS lectures.

http://freescienceonline.blogspot.com/2008/03/more-computing-video-lecture-courses.html

Election criteria

One of my favorite sayings: "Smart people tend to hire people smarter than themselves, dumb people tend to hire people dumber than themselves".

How does this work in the US? Let's see...

• We've got Bush for 2(!) terms.


• Elitist is a dirty word in politics.


• Incidentally, we've got Bush because Kerry was an elitist.


• The other reason we've got Bush is because a whole bunch of morons thought that Bush is more like them which is probably true.


• Hillary Clinton won Pennsylvania primary because voters without college education thought she is more like them. Which is both right and wrong - she's most certainly not like them, but she will definitely pander to them.


• There are no scientists in US Senate.


• But the most powerful Republican at the apex of Republican power in Congress was an exterminator. Yes: as in killing roaches.


• As a result, when confronted with statistics that clearly show that abstinence-only sex education in school does not work, a Republican congressman declares statistics "elitist".


• One third of US teenagers drops out of high school. Another third is not prepared for the next stage of life — either productive work or some form of post-secondary education".


• A quarter of US teenagers cannot identify Adolf Hitler. Large percentage of them think that US was allied with Germany against Russia during WWII. Both trends a boon to neocons because calling someone a "new Hitler" gets quick results from the dumbed down populace.


• Incidentally, there's probably not another place on earth where "neoCON" would be a legitimate name for a political current, not a stigma.


• Especially since they have actually perpetrated a classic Nigerian Scam leading to the Iraq War.

I think THE prerequisite for an effective democracy is an educated demos. This we definitely do not have.

US Aid to Israel

A few interesting numbers here: http://www.wrmea.com/html/us_aid_to_israel.htm

From Bill Maher

"If you think that Democrats are going to take your Bible away, you're an idiot.
If you think that they're going to take your gun away, you're an armed idiot.
And if you think that they're going to take your gun and give it to Mexicans to kill your god, you're Bill O'Reilly"

STL strings

So, while reviewing someone's code, I ran into a place where it was possible to eliminate an unnecessary string assignment. Which got me thinking - is mentioning this even worth it? How expensive is a string assignment, anyway?

So I wrote this simple program:

#define _SECURE_SCL 0

#include <string>

using namespace std;

void f(string a, string *b) {
  *b = a;
}

int main(int argc, char **argv) {
  string y = argv[0];
  string x;
  f(y, &x);
  return 0;
}

and stepped through it in the disassembler. The function f itself was inlined, so I only counted... prepare... the instructions in this call:

call        dword ptr [__imp_std::basic_string<char,std::char_traits<char>,std::allocator<char> >::operator= (402048h)] 

I skipped the call to new (argv[0] turns out to be 69 bytes, just above the default string buffer, so it does allocate), and memcpy itself (memcpy_s to be exact). The net result was - 270+ instructions. If there is no allocation, it is ~150!

This is correct - anywhere between 150 and 270 instructions of STL goo per string assignment IN ADDITION TO ACTUALLY DOING THE WORK!

I tried to do the same on Linux, and after 70-something ddd hung disassembling one of the functions...

Server 2008, first impressions

Short version

MUCH better than Vista.

Medium version

If you're a Microsoft employee and the goons from ITG(*) are trying to rip your favorite OS from your cold, dead hands and make you run Vista, don't - use Server 2008 instead.

There's no LUA/UAC/this idiotic thing that asks you whether you really intend to do what you just asked your computer to do. ipconfig works in a normal shell, instead of demanding one with elevated priorities. It takes literally half as many clicks/keystrokes to do everything.

There are manifestly fewer bugs. It did not fall apart within the first couple of weeks of use. And it seemed much snappier than Vista (although I did not do any real performance benchmarks).

Here is a blog post that has instructions on how to configure Server 2008 into a workstation: http://blogs.msdn.com/vijaysk/archive/2008/02/11/using-windows-server-2008-as-a-super-desktop-os.aspx..

And this one claims that it's 20% faster than Vista, but does not give details on how this number was arrived at: http://vista.blorge.com/2008/03/11/windows-server-2008-is-20-faster-than-vista/.
----
ITG - Microsoft's Informational Technology Group. Changed names multiple times over the years, but its essense stayed the same - these are the people who prevent developers from doing their jobs in the name of security. But this probably merits a separate post...

Long version

I have a lot of storage in my house - not counting client computers, there are approximately 9TB of redundant, RAIDed usable (after RAID) space. Most of my data is on this storage - software, music, pretty much everything.

Every time I buy anything, I immediately copy it to the server, and put the originals in a big box in the basement, where they will stay until the time when BSA people show up at my door and demand the proof of license :-).

My own data is replicated to multiple servers - approximately 300GB of home videos and digital photos, plus other, smaller stuff that accumulated over the years.

Storing stuff requires storage. Storing terabytes of things requires redundant storage. Over the years, I sampled a few RAID-5 solutions, both at home and at work.

There are two major problems with RAID controllers.
(1) While they protect you from a disk failure, they do not protect you from the failure of the controller itself.

Since all of these controllers use proprietary information to describe the RAID array that they store on the disks themselves, they are not interchangeable - you can't take a bunch of disks that you used in RAID mode on LSI and move them to Adaptec (while preserving that data that's on them, that is).

In fact, there's no guarantee that you can move disks between the controllers from the same manufacturer. Or between controllers with different versions of firmware. Or...

So 5 years from now when the RAID card fails, one can very easily be stuck with trying to find an exact replacement of a controller that had been out of production for the last 4.5 years. eBay, anyone?

(2) Software that accompanies these cards is often crappy.

The UI is almost always some atrocious Java program obviously written by a contractor in 2 days right before the product shipped, rife with misspellings and terrible English usage.

I have had multiple problems where midrange cards corrupted data when used on machines with more cores than the manufacturers originally expected.

So unless you test the disk failure scenario right upfront, BEFORE you get any data on it, you may well discover that (a) either the recovery mode does not work, or (2) because of unobvious UI you did something that wiped your disks instead of recovering them.

And good luck finding drivers when you upgrade to the new OS. And since you can't easily move the disks to a new controller... see (1).

Luckily, Microsoft Server family has software RAID subsystem (I am sure Linux has something similar, but coming from Microsoft, I am more familiar with Windows software).

To use it, you have to make your disks dynamic, then you can combine multiple disks into RAID-0, 1, or 5. Volumes of different types can share the same set of physical disks, so for example you can have part of disks 1 and 2, and whole of disk 3 to contribute to RAID-5 volume, and remainder of disks 1 and 2 to form RAID-0 temp storage.

The advantage of soft RAID is that it's hardware independent. You can take all these disks, shove them into any other computer (running Server), and it will still be a RAID volume with all your data intact. The same disk packs can be played on both Server 2003 and Server 2008.

It's an insanely cool idea. Unfortunately, in Server 2003, it was coupled with atrocious implementation.

Soft RAID-5 on Server 2003 is slow. Glacially slow. On my servers that feature dual Xeon 5130s (4 cores per server), with a very decent server motherboard (Supermicro X7DVL-E), and fairly decent mid-range SATA controllers, it was barely doing 20MBps writes, and sometimes would drop to 10MBps for extended periods of time. That on disks that are individually capable of 300MBps transfer rate.

RAID-5 works by partitioning disks into chunks, and then combining chunks from N disks to get N-1 chunk worth of data and 1 chunk of parity. Which means that to write a single sector to a volume, the RAID would have to read corresponding chunks from N disks, compute the parity, and write 2 sectors - one data and 1 parity.

A really terrible implementation would not cache the results of this read, so if the next sector needs to be written, it would repeat all the operations anew, instead of reusing the results of the previous reads.

The only way I can explain the RAID-5 write speed on Server 2003 is that it was this very terrible implementation, although I don't know for sure - an alternative explanation is that maybe they had sleep cycles in there :-).

So when Server 2008 came out, I could not wait to install it and check out its soft RAID implementation. I installed it first on my media server, and then on my data server.

Overall, I was quite impressed. Of course my expectations were very low to begin with because of Vista, but this thing was closer to Server 2003 than it was to Vista. I hit a few bugs right upfront - it hard hung once within a couple of days of installation, and then lost a set of disks (but recovered after reboot).

I am not quite ready to blame it on server itself though, because I added an unknown RAID controller to the machine, and it is more than likely that buggy drivers are to blame. The second server which did not have that controller did not (yet) exhibit this behavior.

Since then, it was relatively quiet and everything functioned the way it supposed to.

The drivers for SATA controllers from Server 2003 worked on Server 2008. The chipset drivers for the motherboard did not. I found that the chipset support for Server 2008 is still quite scarce.

What is unquestionably a bug in Server 2008 is that on RAID volumes the performance counters for logical disks are completely broken - everything except the disk free space and idle time is 0 when it is reading or writing full speed.

But most importantly, its soft RAID implementation is way faster than Server 2003. I copied a few terabytes of data so far, and on writes it does sustained throughput of ~80MBps - 4 times faster than the peak performance Server 2003 could muster. The reads (comparing files between two servers) almost saturate 1GBps network.

So far this things gets my stamp of approval :-).

I am yet to see if it is has long-term stability to last between Windows Update reboots (just in case, I preserved the original installations of Server 2003). I will report on this in a couple of months if everything goes well, earlier if it does not.

Henry Blodget agrees with me...

In today's post on Huffington Post he's basically saying the same thing I wrote about here: http://1-800-magic.blogspot.com/2007/12/risk-vs-reward.html: the CEO comp structure in the US encourages reckless, irresponsible attitude to the long-term shareholder value.

And he's very authoritative on the subject of irresponsible behavior :-)!

Who is flying this plane?

A while ago when I was in Technology Management MBA @ UW, they got former Qwest CEO Joseph Nacchio to come to the orientation meeting and talk about his tenure with the company.

As a result of this talk, I've gotten two very lasting impressions. First was a sense that under absolutely no conditions save outright starvation I would want to work for Qwest, and second was a state of bewilderment to which extent a CEO of the company can be disconnected from the technology that this company is using.

During his talk Nacchio confused GPRS, GSM, and other cell network terms. He professed the ignorance of how the networks work himself, and with a distinct sense of smugness (as in - you don't need to know any of this crap to be a leader - this is for lowly engineers to understand).

What he did talk about at length, and where he had become really animated and very lucid was mergers and acquisitions. Apparently, at the time Qwest was making a lot of money by buying small Eastern European telecoms and then selling them at a profit. From what I gathered in the talk, that line of business was what really animated the management. Not the pesky technological and operational hurdles of providing telecom services.

Over the last many years, as Ballmer's influence increased, I watched Microsoft leadership become less and less engineers, and more and more sales and marketing people.

Now, I don't think that engineering skills are required to head an engineering company. However, I do find that culturally marketing and engineering organizations are VERY different, almost stereotypically, Scott Adams, cartoonish different.

And nothing affects company's culture more than the CEO.

So when I ran into this "pearl" today, I felt embarrassed, but, sadly, not surprised:



There is often a big disconnect between market's perception of the product, and the sales perception it. It is not bad - to be an effective salesperson, you have to be excited about the product as it is - or you won't sell very many of it. Sadly, I fear that Microsoft internal perception - that of the leadership team, anyway - of Vista is closer to the movie above, and not to the painful reality.

Google blog post about the incidents

Here's the post on the Google Lat Long blog about the traffic incidents:
http://google-latlong.blogspot.com/2008/04/new-ways-to-beat-traffic-with-google.html.

Google Road Traffic Incidents ahoy!

I have not posted for a long, long while because the confluence of several events has been keeping me busy almost around the clock for the last 5 weeks.

First and foremost, the project I've been working for the last 3 months - the road traffic incidents - has shipped today. It's my first experience building the entire pipeline at Google - downloading the data from 3rd party provider, parsing it, storing the results in a bigtable, writing a spatial index for it, and, finally, publishing it through the maps frontend server. Here's the result:


It's actually kinda fun watching the data in various cities. Seattle is a really, really quiet place. The incidents are mostly construction (there are a few accident icons, but in reality they are all about slow traffic):


Though Seattle is a small place, actual real accidents (as in - collisions) are quite uncommon even in much bigger cities. Here's New York. Here you see a lot of road closures (most of them are periodic, and in the future - eventually we'll figure out how to be more intelligent about displaying them; right now the data we're getting is lacking a lot of details that would allow us to be - but we'll work it out with our provider). There are a few constructions, and slowdowns, but only one of the icons is displaying a collision:


And most other US cities are somewhere between New York and Seattle. Except for one. If you're living in LA, my hat is off to you. The traffic there is real zoo. Yes, there's even one place where almost everyday the sign shows "Animals on the road". Here's a quiet night at LA (believe it or not, during the day time, it's a lot worse). Most non-consturction icons are real collisions. A couple are hit and runs.


The insane amount of construction icons you see here is a nightly event - the authorities deply miriads of crews to do various road maintenance tasks on the roads in the evening, and it lasts through the night. And they report a construction incident for every one of them. During the day, the roadwork in LA clears, leaving mostly just collisions. But a lot of them!

Adam Smith on wars...

"In great empires the people who live in the capital, and in the provinces remote from the scene of action, feel, many of them, scarce any inconveniency from the war; but enjoy, at their ease, the amusement of reading in the newspapers the exploits of their own fleets and armies. To them this amusement compensates the small difference between the taxes which they pay on account of the war, and those which they had been accustomed to pay in time of peace. They are commonly dissatisfied with the return of peace, which puts an end to their amusement, and to a thousand visionary hopes of conquest and national glory from a longer continuance of the war."

-- An Inquiry into the Nature And Causes of the Wealth of Nations

Dr. Strangelove... in reverse!

President Bush rarely comments about the Democratic presidential contest, but he said that he had to speak up about Clinton's red phone ads because he found them "so confusing."

"If I answered the red phone every time it rang, I would never get any sleep," Bush said. "Sometimes it starts ringing at 9 p.m., and I am already tucked in by then."

Bush said that "there's nothing so important that it can't wait until tomorrow, or whenever I remember to check my voicemail."


http://news.yahoo.com/s/uc/20080308/cm_uc_crabox/op_475477;_ylt=AmW7uqcqRkkMlQ.hJ9SIda9xFb8C

This is a loop that never ends - it just goes on and on my friends!

    for (double d = 0.0; d != 1.0; d += 0.1)
      System.out.println(d);

:-)

10,000 BC: a movie review

Short version: terrible.

Medium version: think 300. This movie is very similar to it in terms of both connection to reality and the overall flow.

Long version (WARNING: a plot spoiler): there's this really advanced civilization in the middle of desert 12000 years ago. They know metalworks (and it looks like iron, too), shipbuilding, navigation, astronomy, writing, cloth manufacture. Basically, Atlantis, except in Africa. They build huge gold-tipped pyramids, elevated roads, and babylonian-looking temples and are rulled by a priest class reporting to chief priest AKA the "Almighty". To build all this they need, obviously, slaves, and so they send the roaming bands to snatch them from neighboring tribes.

The neighbors are more authentically looking 10000BC-ers (at least they wear fur, and appear to not use iron weapons). The only problem - their men are either shaved, or have neatly trimmed beards and goatees.

Anyways, one of these roving bands steals this good-looking girl, and it proves to be their undoing: her boyfriend comes to rescue her, incites the global slave revolt, easily overcomes the few armed (iron swords!) guards that are there. In the thick of the fight the gilfriend gets shot, but then is magically revived by a magician thousands of miles away.

Oh, yes, and at the end of all this our hero's tribe gets a bag of magic beans and starts the agricultural revolution. Viva Monsanto!

Java collections perf redux

I finally got some time to run my microbenchmark for various Java collection objects.

Here's the source:

package com.solyanik.perftesting;

import java.util.Arrays;
import java.util.Vector;
import java.util.ArrayList;

/**
 * Implements a micro-benchmark using
 * high-resolution RDTSC timer.
 * 
 * @author sergey@solyanik.com (Sergey Solyanik)
 */
public class MicroBenchmark {

  /*
   * RDTSC - returns number of clocks since
   * the core was last reset.
   * 
   * NOTE: no synchronizing instruction
   * is executed before RDTSC, so this
   * is not suitable for really short
   * runs as the second RDTSC can execute
   * before the instruction we're trying
   * to measure.
   * 
   * NOTE2: Different cores have
   * different clock counters, and they
   * are not synchronized.
   */
  private native static long rdtsc();
  
  /**
   * A function that consists of one
   * instruction - ret.
   */
  private native static void nop();
  
  /**
   * Fixes the thread to the first CPU
   * and core.
   */
  private native static void fixthread();

  /**
   * Loads the library containing our
   * native code.
   */
  static {
    System.loadLibrary("jnipc");
  }
  
  /**
   * Prints the statistics for an array
   * of clock intervals.
   * 
   * @param clocks The array of clocks
   * @param iterations Number of iterations
   * @param bucket_number How many buckets
   *     to use for the histogram
   * @param scale Scale for the histogram
   */
  private static void Analyze(
      String header, long [] clocks,
      int iterations, int bucket_number,
      int scale) {
    Arrays.sort(clocks);
    double average = 0.0;

    long [] buckets = new long[bucket_number];

    double log2 = Math.log(2.0);

    for (int i = 0 ; i < clocks.length; ++i) {
      average += clocks[i];
      int ndx = (int)(Math.log(
          (double)clocks[i]
          / (double)scale) / log2);
      if (ndx < 0)
        buckets[0]++;
      else if (ndx < buckets.length - 1)
        buckets[ndx + 1]++;
      else
        buckets[buckets.length - 1]++;
    }

    average /= clocks.length;

    System.out.println(header + ":");
    System.out.println("Median for " +
        iterations + " iterations: " +
        clocks[clocks.length / 2]);
    System.out.println("Average for " +
        iterations + " iterations: " +
        average);
    System.out.println("Median per iteration: "
        + clocks[clocks.length / 2]
                 / iterations);
    System.out.println("Average per iteration: "
        + average / iterations);

    System.out.println("Histogram:");    
    int prev = 0;
    int curr = scale;
    for (int i = 0; i < buckets.length - 1; ++i) {
      System.out.println(prev + " - " + curr
          + ": " + buckets[i]);
      prev = curr;
      curr *= 2;
    }

    System.out.println(" > " + prev + ": " +
        buckets[buckets.length - 1]);    
  }
  
  /**
   * Number of loop iterations per sample.
   */
  private static final int ITERATIONS = 10000;
  
  /**
   * Number of samples.
   */
  private static final int SAMPLES = 1000;
  
  /**
   * Number of buckets for the histogram.
   */
  private static final int BUCKETS = 20;
  
  /**
   * Bucket scale for the histogram.
   */
  private static final int BUCKET_SCALE = 1000;
  
  /**
   * Main function.
   * 
   * @param args
   */
  public static void main(String[] args) {
    fixthread();
    System.out.println("Let the fun begin!");
    
    long [] clocks = new long[SAMPLES];

    Boolean b = new Boolean(true);

    for (int i = 0; i < clocks.length; ++i) {
      Boolean [] a = new Boolean[ITERATIONS];
      long start = rdtsc();
      for (int j = 0; j < ITERATIONS; ++j)
        a[j] = b;
      long stop = rdtsc();
      clocks[i] = stop - start;
    }
    
    Analyze("Array",
        clocks, ITERATIONS, BUCKETS,
        BUCKET_SCALE);
        
    for (int i = 0; i < clocks.length; ++i) {
      Vector<Object> v = new Vector<Object>();

      long start = rdtsc();
      for (int j = 0; j < ITERATIONS; ++j)
        v.add(b);
      long stop = rdtsc();
      clocks[i] = stop - start;
    }

    Analyze("Expanding Vector", clocks,
        ITERATIONS, BUCKETS, BUCKET_SCALE);

    for (int i = 0; i < clocks.length; ++i) {
      Vector<Object> v = new Vector<Object>();
      v.setSize(ITERATIONS);
      long start = rdtsc();
      for (int j = 0; j < ITERATIONS; ++j)
        v.set(j, b);
      long stop = rdtsc();
      clocks[i] = stop - start;
    }

    Analyze("Pre-allocated Vector",
        clocks, ITERATIONS, BUCKETS,
        BUCKET_SCALE);

    for (int i = 0; i < clocks.length; ++i) {
      ArrayList<Object> al = new ArrayList<Object>();
      long start = rdtsc();
      for (int j = 0; j < ITERATIONS; ++j)
        al.add(b);
      long stop = rdtsc();
      clocks[i] = stop - start;
    }

    Analyze("Expanding ArrayList", clocks,
        ITERATIONS, BUCKETS, BUCKET_SCALE);

    for (int i = 0; i < clocks.length; ++i) {
      ArrayList<Object> al = new ArrayList<Object>();
      al.ensureCapacity(ITERATIONS);
      long start = rdtsc();
      for (int j = 0; j < ITERATIONS; ++j)
        al.add(b);
      long stop = rdtsc();
      clocks[i] = stop - start;
    }

    Analyze("Pre-allocated ArrayList", clocks,
        ITERATIONS, BUCKETS, BUCKET_SCALE);
  }
}

And here's what I've got:

                     Median Average
Array                    14      15
Expanding vector        124     133
Pre-allocated vector    115     119
Expanding ArrayList      57      66
Pre-allocated ArrayList  43      46

To remind, C++ numbers were like this:

                     Median Average
Array                     2       2
Pre-allocated vector,
  assigment               2       2
Pre-allocated vector,
  push_back               8       8
Expanding vector,
  push_back             145     148 (*)

(*) Obviously, this number very much depends on number of expansions, because STL vector grows by 50%, a very big vectors will have smaller cost. See my previous posts for details on C++ testing.

Synchronization in a web application

UPDATED 03/08/2008 - source code completely rewritten.

While I was writing the code for the Project Guttenberg proxy, one of the problems I needed to solve was the concurrency in URL processing.

When a user requests a URL from the proxy, the proxy sends the request to the real Project Gutenberg web site, and writes the result into a file.

Then it processes the file - if it's an HTML document, it replaces the references to the original web site with links that point to the proxy. If it was a request for a LRF file, the text file that was fetched from the Project Gutenberg would be translated.

Then there's cache - the web pages that have already been fetched, but have not expired get fetched from that cache.

In other words, there's a lot of activity, that is by and large independent - unless the two requests come for the same page. Then of course we do not want to process them at the same time, because they are going to be modifying the same state, corrupting each other in the process.

The solution is not that hard, but I thought it would be instructive to put it here.

package com.solyanik.sync;

import java.util.HashSet;

/** The general object lock class.
 * Ensures that any two objects
 * that are equal to each other as
 * in a.equals(b) cannot be locked
 * simultaneously.
 * 
 * This is a complement to the
 * standard monitor functionality
 * that protects objects that are
 * equal in == sense (i. e. the
 * same object).
 *
 * IMPORTANT NOTE: The same thread
 * trying to lock the same object
 * twice = INSTANT DEADLOCK!
 * 
 * Usage:
 *    // in constructor
 *    ObjectLock lock = new ObjectLock();
 *    ... 
 *    void handleRequest(final Object o) {
 *      try {
 *        lock.lock(o);
 *        ... // do processing
 *      } finally {
 *        lock.unlock(o);
 *      }
 *
 * @author sergey@solyanik.com (Sergey Solyanik)
 */
class ObjectLock {
  /**
   * Stores all currently locked objects
   */
  private HashSet<Object> locked =
      new HashSet<Object>();
  
  public void lock(Object o)
      throws InterruptedException {
    synchronized(locked) {
      for ( ; ; ) {
        if (locked.contains(o)) {
          // We are already locked.
          locked.wait();
        } else {
          locked.add(o);
          break;
        }
      }
    }
  }
  
  public void unlock(Object o)
      throws IllegalMonitorStateException {
    synchronized(locked) {
      if (!locked.remove(o))
        throw new
            IllegalMonitorStateException(
            "The object is not locked.");
      locked.notifyAll();
    }
  }
}

The code is tiny and by and large self explanatory. We keep a hash set, and whenever we are about to lock the object, we check if it is in the hash set already. If it is not, we put it there and continue on our merry way.

If it is, we're joining the queue of threads waiting on the hash set (it can actually be absolutely any object). When the object is unlocked, it will be removed from the hash, and the threads waiting will be notified.

To preempt obvious question - this assumes that the level of contention is not super high, so there are not a lot of threads pending at any given time. In the case of a high level of contention (10s of threads), we would want to wake up only the threads that are waiting for that specific object that has just been unlocked, not all of the threads.

How is this different from synchronized(o)? The ObjectLock will prevent collisions of code operating on objects that are equal in the sense that o1.equals(o2). The synchronized clause prevents operations on the objects that are the same (o1 == o2).

In the case of a string, by the way, an inefficient way of doing the same that the lock class above is trying to accomplish is this:

  void handle_request(String url) {
    String lock = url.toLowerCase().intern();
    synchronized(lock) {
       ... do processing ...
    }
  }

Interned string have the property of s1 == s2 iff s1.equals(s2). The problem of course is that every url that has ever been processed will stay in your process's string heap forever, so it is not very efficient.

And the unit test:

package com.solyanik.sync;

import java.util.HashSet;

public class UnitTest {
  static class Executor extends Thread {
    private static HashSet<String> inuse
        = new HashSet<String> ();

    private String url;
    private int id;
    private ObjectLock lock;
    
    public Executor(String url, int id,
        ObjectLock lock) {
      this.url = url;
      this.id = id;
      this.lock = lock;
    }

    public void run() {
      try {
        System.out.println("[" + id +
            "] Started to process " + url);
        System.out.println("[" + id +
            "] Locking " + url);
        lock.lock(url);
        System.out.println("[" + id +
            "] Lock acquired " + url);
        // Yes, I know this is unnecessary.
        // But just to be absolutely sure,
        // this is a test after
        // all...
        String internedUrl = url.intern();

        if (inuse.contains(internedUrl)) {
          System.out.println("[" + id +
              "] ERROR: COLLISION " + url);
          throw new AssertionError(
              "Bad lock implementation!");
        }

        inuse.add(internedUrl);
        Thread.sleep(5000);
        inuse.remove(internedUrl);

        System.out.println("[" + id +
            "] Releasing " + url);
        lock.unlock(url);
        System.out.println("[" + id +
            "] Lock released " + url);
        System.out.println("[" + id +
            "] Thread finished ");
      } catch (InterruptedException ex) {
        System.out.println("[" + id +
            "] Interrupted: " + ex);
      }
    }
  }
  
  /**
   * @param args Not used.
   */
  public static void main(String[] args) {
    String [] urls = {
       new String("http://www.google.com"),
       new String("http://www.google.com"),
       new String("http://www.google.com"),
       new String("http://www.google.com"),
       new String("http://www.google.com"),
       new String("http://www.live.com"),
       new String("http://www.live.com"),
       new String("http://www.live.com"),
       new String("http://www.yahoo.com"),
       new String("http://www.yahoo.com")
    };
    ObjectLock lock = new ObjectLock();
    for (int i = 0; i < 128; ++i) {
      (new Executor(urls[i % urls.length],
          i, lock)).start();
    }
  }
}

JNI Made Simple

I want to measure the performance of Java vector to compare it with STL. Java of course lacks a really high resolution timer, so I needed to use JNI - the Java native interface.

After reading Sun's spec here: http://java.sun.com/j2se/1.5.0/docs/guide/jni/spec/jniTOC.html, I decided that there needs to be a simplified version. So here it goes. A simple, 3-step recipe.

(1) Create a DLL. Here's an example of a DLL that contains one function that does nothing:

nop.cxx:

__declspec(naked)
unsigned __int64 __cdecl nop(void) {
  __asm {
    ret
  }
}

nop.def:
LIBRARY "nop"
EXPORTS
  Java_Test_nop = nop

Notice the decorated name. Java_ is required for all, Test is the name of the class, "nop" is the name of the function. Straightforward.

(2) Compile this using Visual Studio, and copy the resulting nop.dll to the root directory of your Java project.

(3) Call it from Java:

class Test {
  native static void nop();
  
  static {
    System.loadLibrary("nop");
  }

  public static void main(String[] args) {
    nop();
  }
}

That's all there is to it. Let's now look at how fast the function call is actually made. To do this, I am going to use the same framework I used for testing STL's vector performance: http://1-800-magic.blogspot.com/2008/02/stl-vector-performance.html, except translated to Java.

We'll need 3 functions in the native library though - one that does nothing (this is the function we're going to test), the other will be returning the results of Pentium's RDTSC instruction that measures the clock count, and the third one will be fixing the thread's affinity to the first core, because if this is not done and thread jumps the core while executing, we won't be able to compare the results of RDTSC, since clock counters on multiple cores is not synchronized.

Here's how the native code looks like:

jnipc.cxx:
#include <windows.h>

__declspec(naked)
unsigned __int64 __cdecl rdtsc(void) {
  __asm {
    rdtsc
    ret
  }
}

__declspec(naked)
unsigned __int64 __cdecl nop(void) {
  __asm {
    ret
  }
}

void __cdecl fixthread(void) {
  SetThreadAffinityMask(GetCurrentThread(), 1);
}


jnipc.def:
LIBRARY "jnipc"
EXPORTS
  Java_Test_rdtsc = rdtsc
  Java_Test_nop = nop
  Java_Test_fixthread = fixthread

The Java code is more involved, but less complicated :-):

import java.util.Arrays;

class Test {
  native static long rdtsc();
  native static void nop();
  native static void fixthread();
  
  static {
    System.loadLibrary("jnipc");
  }
  
  static final int ITERATIONS = 100000;
  static final int SAMPLES = 1000;
  static final int BUCKETS = 20;
  static final int BUCKET_SCALE = 10000;
  
  public static void main(String[] args) {
    fixthread();
    long [] clocks = new long[SAMPLES];
    for (int i = 0; i < SAMPLES; ++i) {
      long start = rdtsc();
      for (int j = 0; j < ITERATIONS; ++j)
        nop();
      long stop = rdtsc();
      clocks[i] = stop - start;
    }

    Arrays.sort(clocks);
    double average = 0.0;

    long [] buckets = new long[BUCKETS];

    double log2 = Math.log(2.0);

    for (int i = 0 ; i < SAMPLES; ++i) {
      average += clocks[i];
      int ndx = (int)(Math.log((double)clocks[i]
          / (double)BUCKET_SCALE) / log2);
      if (ndx < 0)
        buckets[0]++;
      else if (ndx < BUCKETS-1)
        buckets[ndx + 1]++;
      else
        buckets[BUCKETS-1]++;
    }

    average /= SAMPLES;

    System.out.println("Median for " +
        ITERATIONS + " iterations: " +
        clocks[SAMPLES / 2]);
    System.out.println("Average for " +
        ITERATIONS + " iterations: " +
        average);
    System.out.println("Median per iteration: "
        + clocks[SAMPLES / 2] / ITERATIONS);
    System.out.println("Average per iteration: "
        + average / ITERATIONS);

    System.out.println("Histogram:");    
    int prev = 0;
    int curr = BUCKET_SCALE;
    for (int i = 0; i < BUCKETS - 1; ++i) {
      System.out.println(prev + " - " + curr
          + ": " + buckets[i]);
      prev = curr;
      curr *= 2;
    }

    System.out.println(" > " + prev + ": " +
        buckets[BUCKETS - 1]);

  }
}

On my T2600 Core 2 Duo laptop this ran in approximately 38 clocks per iteration.

Median for 100000 iterations: 3815903
Average for 100000 iterations: 3854832.709
Median per iteration: 38
Average per iteration: 38.54832709
Histogram:
0 - 10000: 0
10000 - 20000: 0
20000 - 40000: 0
40000 - 80000: 0
80000 - 160000: 0
160000 - 320000: 0
320000 - 640000: 0
640000 - 1280000: 0
1280000 - 2560000: 0
2560000 - 5120000: 998
5120000 - 10240000: 2
10240000 - 20480000: 0
20480000 - 40960000: 0
40960000 - 81920000: 0
81920000 - 163840000: 0
163840000 - 327680000: 0
327680000 - 655360000: 0
655360000 - 1310720000: 0
1310720000 - -1673527296: 0
 > -1673527296: 0


C:\bin>java -version
java version "1.6.0_03"
Java(TM) SE Runtime Environment (build 1.6.0_03-b05)
Java HotSpot(TM) Client VM (build 1.6.0_03-b05, mixed mode, sharing)

To compare it with C, I've added the following case to my perf program in http://1-800-magic.blogspot.com/2008/02/stl-vector-performance.html:

case 4:
  printf("Function call\n");
  HMODULE hm = LoadLibraryW(L"c:\\bin\\jnipc.dll");
  void (*func)() = (void (*)())
    GetProcAddress(hm, "Java_Test_nop");
  for (int i = 0; i < NUM_SAMPLES; ++i) {
    unsigned __int64 begin = rdtsc();
    for (int j = 0; j < NUM_ITERATIONS; ++j)
      func();
    unsigned __int64 end = rdtsc();
    clocks[cptr++] =
        (unsigned int)(end - begin);
}

...and got approximately 6 clocks per call, making JNI about 6 times slower on this simple test:

Function call
Median: 60112
Median, CPI: 6
Average: 60262
Average, CPI: 6
Histogram:
0 - 1000: 0
1000 - 2000: 0
2000 - 4000: 0
4000 - 8000: 0
8000 - 16000: 0
16000 - 32000: 0
32000 - 64000: 998
64000 - 128000: 1
128000 - 256000: 1
256000 - 512000: 0
512000 - 1024000: 0
1024000 - 2048000: 0
2048000 - 4096000: 0
4096000 - 8192000: 0
8192000 - 16384000: 0
16384000 - 32768000: 0
32768000 - 65536000: 0
65536000 - 131072000: 0
131072000 - 262144000: 0
 > 262144000: 0