I did a “Hello World” program which got me a verification: sorting 100 million random integer on a 15 GB JVM.  The version with –XX:+CompressedOops ran 181 sec, over 205 sec without, i.e. 12 % faster.
In summary, I think it is a good idea to have compressed object pointers always on by default, if you need less than 32 GB.

ExecutorService in combination with BlockingQueue, see the JavaDocs for instance.

The idea is to decouple the invoker thread from the worker thread. The invoker dumps a task onto a blocking queue and returns immediately. Any available worker thread picks a task off the queue and does its thing. This sort of decoupling makes for a great user experience in the UI layer and minimal chance for thread contention in a multi-threaded environment. This is how I would  implement log4j, if it isn’t already doing this, or asynchronous JMS messaging.

There is even a DelayQueue since Java 6 which allows to poll objects of a minimum age. This can be used as a lightweight solution for aggregation in memory (no ESB, JMS, DB, etc. necessary) if all the data fits into one JVM.

Also on a side note, prefer using an ExecutorCompletionService over a simple InvocationService. The latter runs only as fast as it’s slowest running task, while the first one runs tasks continously off a queue with never a thread in the pool idle.

Lastly, the book Java Concurrency in Practice by Brian Goetz is a must for Java Developers. What a great book!

This is where I had the idea for this following metaphor: Contractors in Software Engineering do the same sometimes, i.e. they leave behind a mess. They get in, do something and get out. From a pure survival point of view I understand this. But what about the mess over our head? When does an asset become a liability? When does the rot harm vital functions of the house?

Those starting a new house (project) from scratch should pause now and think about it. It is always easier to keep the attic clean and organized from the beginning.

Returning to my metaphor, my advice is to not allow for a deadzone in a project. Everything in a project should be exposed to the fullest to disallow deadzones. CM tools like Maven, unit test coverage reports (Cobertura), Hudson automated builds and visualizations with Sonar, all help to make a project transparent and detect outliers early. I think the first week for anybody new on a project should include training in the quality standards of the organization. Those need to be kept alive, updated, revised and evolved by a few select veterans with interest and passion in SWE. So anyone  on a SWE project, regardless if the employment status is permanent or contractor, is to be held accountable by the common standards. If somebody says his code is done, but it does not pass the internal quality standards, he or she is not done by definition.

They say praise in public, criticize in private. However, let me add a warning to this. Praising in public may become a boomerang if (i) the person praised does not want to be in the spotlight, and (ii) the others feel as if they are criticized at the same time.

Back to the technique of praising as far as words are concerned. Don’t use Great Job! all the time. It wears off fast and after a while it sounds like you say Good Dog! To be more genuine use different phrases. My wife showed me a great website named 101 Ways to Praise, see http://www.gesd40.org/mainsite/parent/pghtm/101ways.htm. It is geared toward child education, and a few quotes may be inappropriate at work. However, most of them fit perfectly. I keep a color printout of this page in my drawer. The quotes are funny and creative, and really helpful for effective communication.

But since Java 5 annotations, there is also JSR-303 Bean Validation API, which may make it into Java 7. For an intro e.g. see http://musingsofaprogrammingaddict.blogspot.com/2009/02/getting-started-with-jsr-303-bean.html

For example a declaration of this member variable

    @NotNull

    @Size(min = 2, max = 14)

    @UpperCase

    private String licensePlate;

requires an instance of licensePlate to be not null, between 2 and1 4 chars long and all uppercase. Isn’t that awesome? Countless examples come to mind where this would have been helpful? Application data usually ends up in a database or on a UI screen and always needs to be limited in some or another way. Think of the example of US social security numbers, or an Oracle varchar(4000) as the largest data type for a String. JSR-303 covers a really basic need that has been missing in Java since the beginning, IMO.

So don’t implement your own, use the Hibernate reference implementation for instance, see http://hibernate.org/subprojects/validator, which is not only for the database tier. They specifically say that it can be used for the web tier or Swing UI, etc, wherever you have a data validation need.

Pros

• fast (instant) allocation of computing resources.
• fast prototyping, fast product development.
• shortened business cycle, shortened time to market.
• scalable: no lead time in building out data center.
• reduced cost of hardware and staff.
• reliability, high uptimes and SLAs, or else the cloud providers risk their reputation
• new business models perhaps. E.g. when the volume and bandwidth of data and its geographic distribution creates a new opportunity that could not be served out of a single data center.
• global applications can be distributed closer to end users, application latency is smaller, users are happier.

Cons

• additional hard dependency on outside party, the cloud provider.
• even the best clouds have outages, see http://status.aws.amazon.com/s3-20080720.html, i.e. you probably shouldn’t build a life support or an air traffic control system in the cloud.
• it removes a company capability that may have been essential to the company in the past.
• once the people are gone, training new people, building up IT capacity takes years.
• on the other hand, like any new technology, cloud computing requires young-in-mind technologist.
• privacy concerns for sensitive data: is financial or patient data safe? Is it even legal?
• the new bottleneck becomes the network.
• the WAN is not really reliable, the software to push data to for instance S3 endpoints needs to be a lot more resilient than the code written for local networks (retries, timeouts, throttling, encryption,…).
• depending on the distribution of data and application hosting, additional network latencies may cloud  the end-user experience.

1. The job market for certain IT people will become tighter, kind of like it was in 2002. Businesses will shed more people. They are concerned to stretch their assets as much as possible through the economic downturn.
2. The agile ways of delivering software will be the main stream paradigm. SCRUM, XP, you name it, are now household names in IT.
3. Continuous build systems are ubiquitous, every new keystroke of code triggers a new build in the background.
4. SOA is continuing to grow, but only as much as it is simplified. E.g. REST will be the dominant architecture for web services, mainly because the big ones, Google, Y!, Amazon, offer more and more services in REST. SOAP will persist but shrink in significance because too complicated with too many standards.
5. Opensource frameworks will be a standard part of enterprise IT, it only grows in market share, not because it is free, but because it is leight weight, well tested and cared for by large communities of enthusiastic people. I mean those projects that are successful.
6. On the ESB side Mule and ServiceMix will make the ESB paradigm very successful, not only becausae of all the benefits of an ESB, but also because the community behind it is growing.
7. The purity of relational data models will become less important as more and more processes need to perform really fast on large data sets. The trends is toward simplification of data storage, think of flat files or web services in the cloud.
8. Similarly, network attached memory for JVM clusters (check out Terracotta) is cheaper, faster, and simpler than a large clustered Oracle data warehouse, to process large stateful data sets.
9. Java EE remains the dominant enterprise platform. Tools like Eclipse will deploy code changes automatically and instantly to applications servers in real time (as some plugins already do). Java development will be as convenient as working with scripting languages like Perl or Ruby.
10. Solid-state disks will become the choice for fast computation, magnetic hard disks are the new tape for permanent storage and backups

There would be a few other things like multi-threading and parallelization in enterprise computation, but I am running out of time now.

Let’s see next year how much of that became reality.

Happy New Year everybody!

Okay, the thing was possibly +40 years old and served the kitchen sink drain. A layer of black stinky crap narrowed the section of free flow to the innermost center while corroding away the metal on the outer side. My point here is that software rots too. Not in a physical sense; software is virtual and the 1’s and 0’s don’t change. But the analogy is figurative: software rots because the environment changes, requirements change, people come and go, the demands of reality on a system can change substantially: for instance a higher load, a different use than it was designed for, a third-party changes its rules, the list is endless. In such a situation, a software system can behave like this broken pipe. It cannot do it’s job anymore. There is a philosophical/economical debate as to what is better in that case: the minimal fix until the next crash, or a rewrite from scratch. I don’t want to elaborate on this now. My point is that this photo made me think of software as a fragile ecosystem that degrades constantly because its surroundings (we and our businesses) change constantly.

The picture below shows a memory leak of a Tomcat web application that reads from ActiveMQ messaging system and does a bulk insert into a database. The memory graph, snapshot from the JMX console, rises over 9 hours close to 800 MB before it throws periodic OutOfHeapSpace exceptions. The Tomcat instance was run with -Xmx1024m. Tomcat uses the rest for itself. This measurement was taken under the conditions of a stress test; it levels out at 800 MB after 9 hours. In production, where JMX is turned off, the OutOfHeapSpace error took a week or so to show up.

How to diagnose a memory leak

How to find the root of the leak

In my case, the leak was caused by a JDBC database driver, JConn v6, for reasons I don’t really bother to know. Replacing that JDBC driver with the open-source jDTS driver for Sybase and SQLServer brought down memory consumption to 20MB and made it run faster. A nagging problem was solved.