Moving to a Mac

With my new gig at Talend I have requested a MacBook Pro as a company computer, thinking that since I’m now an architect I will have to do more writing and (god forbid) slides. Of course I will continue to develop software, both for the company and in my open source work. For years, my preferred development platform has been Unix (shows how old I am, I really mean Unix) or Linux. And I have had to work on all three of Windows, Linux and the Mac to test my software. Though I had a Mac, I did not work on it very much. Thanks to Eclipse things just worked there pretty well, I only had a few Mac-specific issues.

So I started to move everything to the Mac and I was shocked at how good it was. Really just good. Everything just feels better than the Ubuntu GUI that I was using (Gnome 3 — I just could not deal with Unity). I spent the money and got the Thunderbolt display and it’s just amazing, so big and clear, and it is essentially a docking station. Everything works fine with my odd keyboard and mouse.

Unfortunately, there is an Eclipse SWT issue (it looks like) which is preventing me from running all 3000 or so of the data transformation unit tests on the Mac, so I have spent a couple of days working on characterizing it, and the thing that I found really shocking about the Mac was Instruments. I could get a ton of profiling information from a running program with an amazing GUI with no effort. By combining this with JProfiler, I should be able to get what I need to characterize (and even possibly fix) this leak problem.

I have never considered having Windows as my daily machine, and now that I have had the Mac, I’m never going back. It’s just that good.

Now at Talend

After many years of leading Oakland Software to develop really good data transformation technology, I have moved to a new phase, having sold these assets to Talend. A little while ago, I joined Talend as a Senior Architect responsible for data transformation as well as other cross product issues and I’m very excited to be part of this team of extremely bright and talented folks.

For quite a while now the Oakland Data Transformer has worked well with the Talend ESB runtime technologies, and now that it’s part of Talend we will work on integration at the Talend Open Studio level for a future Talend release. You will hear more about that as the work progresses.

I am very grateful to the open source community, mostly to my friends at Eclipse on which my product is based. It would not have been possible to make such a high functionality and robust product without the extensive infrastructure provided by Eclipse to do such things. I think however that the most important part of Eclipse is the culture. It’s a culture of openness, mutual respect, and encouragement. People are encouraged to be nice, helpful, and not at all arrogant and this is the case with everyone I have worked with there over the years. It’s been an amazing experience being part of it and I hope to continue my small contributions to Eclipse.

Jenkins EC2 work

About a year ago, I have gotten involved pretty extensively in the Jenkins EC2 plugin where I’m now the maintainer. This was motivated by wanting to move my company’s (Oakland Software — now mostly part of Talend) build process to EC2 and finding out that it was simply not possible unless some significant work was done on the plugin. Seeing that the plugin was missing an active maintainer, I requested the responsibility (which was quickly granted as it the custom in Jenkins).

I have done 3 major releases in the last year with both contributions originated by me and from many members of the community adding significant features like:

1. Support stopping instead of terminating instances
2. Allow the use of EC2 spot instances
3. Proper support for multiple clouds
4. Greatly increase the accuracy and reliability of starting and stopping instances and adhering to limits
5. Many bug fixes

I remain active with this project and working closely with several members of the Jenkins community to improve it. With these improvements the plugin has become more popular gaining nearly 200 new installations in the past year.

In addition to coordinating and helping with contributions, I would like to start a conversation about looking the new Google Compute Engine service, which is similar to EC2. We should be able to leverage what we have learned in the EC2 plugin and to the refactoring necessary to cleanly support the Google service in addition to Amazon’s. Ideas and contributions are welcome.

Some thoughts on the Java u21 problem

I see all of the platform UI bugs and https://bugs.eclipse.org/bugs/show_bug.cgi?id=319514 caught my attention and I have made several comments on it. I thought I would share my observations and suggestions at this point, being a member of the Eclipse community.

Observations:

1. This problem happens when you run Helios on Windows (possibly other platforms) with the current version of Java (Java 6 update 21). The problem also manifests in the worst possible way, it shows up as a hang (or possibly and infinite loop). As I write this, I’m sure that hundreds of hours are being spent trying to figure out what’s going on by people trying to use Eclipse.
2. If there is such a thing as an emergency about things not working right, I would say this would be it. Downloading the current stuff is broken.
3. This is an Eclipse bug. We are using the internals of software that we should not be, it changed under us and it broke our stuff. Also we did not certify with this software. We need to fix this problem. Complaining about that Oracle changed it and depending on them to fix it is a disservice to our users. There are many many examples of this sort of thing happening (mainly in SWT) and in most cases Eclipse seems to do the right thing by working around the problem and fixing Eclipse so that the people using Eclipse have the best experience possible.
4. The bug happens with many previous versions of Eclipse due to the Oracle VM change, and some solution needs to be provided to them.

Suggestions:

1. Fix Helios right now. Today if possible. Yes this will require a 3.6a or whatever, but we have to make this configuration seamlessly work.
2. I know a lot of work has been done to develop workarounds and solutions for the previous releases. These need to be widely communicated, including on the front page of eclipse.org. We need to get the word out to help people. And we have to keep in ming people will be looking for a solution to a hang.
3. Improve our release certification process. There is a Java release process where they test and and provided releases available for testing and certification precisely so that these problems don’t happen. We are one of the leading IDEs for Java, we should be totally on top of this process (including having relationships internally with Oracle so that mechanism are in place such that stuff does not get broken if necessary). We have the means to see this is coming and we should have prepared for it by certifying Helios on the u21 version of Java. I know we are pretty formal about talking about our reference platforms, we need to do better here.
4. Have some policies about accessing the internals of dependent products. Of course it’s necessary to do this in a lot of areas (particularly SWT and Equinox), but where this is done there should be some extra review and also some more centralized tracking so that we understand these dependencies and can consider this in our certification. Also having the extra review will minimize the temptation to quickly add something that’s an internal dependency. People should try to think long and hard about this before doing it to see if there is another way.

This problem is a very serious problem, and we should do like what they do when they investigate airplane crashes, we should look at this from every angle and get as much learning  out of this as possible and implement improvements of the process at all levels to make sure it does not happen again.

MercurialEclipse and Andrei the “handle it” dude

As I posted previously I was having real trouble with hg and MercurialEclipse.

It turns out a lot of my problems with slowness in decorator updating and startup with Eclipse were related to the number of projects (and maybe size) of my repository. The repo contains 114 projects, a total of 3059 folders and about 98000 files. One project is about 500M, two are about 250M and the rest are pretty small, about half of the projects are only a few K (Eclipse feature projects). About 125 of the files exceed 2MB, 4 exceed 5MB and 1 exceeds 10MB.

I reported a bug on this yesterday http://www.javaforge.com/issue/11928 and immediately Ekke (an hg/MercurialEclipse fan) made some tests and posted them there, Andrei (the MercurialEclipse maintainer) asked several questions and now, less than 48 hours after reporting the bug report, I have received two updates of MercurialEclipse, the second of which reduced the start of time from close to 4 minute to 30 seconds, which is in the acceptable realm.

A while ago I had a similar response from Andrei with a showstopper issue for me which was resolved in less than 24 hours.

I am very happy I made the decision to go with hg, even though it’s a relatively new path from a number of perspectives and the main reason I’m happy about it is that I know that any real problems will be taken care of immediately, and that’s something that’s very rare in this business, no matter how much money you are willing to pay.

Thanks Andrei!

Sadly, converting from Subversion to Mercurial *almost* failed (revised)

(Since posting this I’m more encouraged and I am keeping going, in fact I have now committed to the switch — there are some serious performance problems for my large repo, but given the responsiveness of the HgEclipse folks I expect it will be resolved soon, and I am able to get work done — Thanks Ekke for your comments and encouragement about this)

I really loved the idea of using a DVCS and I settled on Mercurial (hg) because I use a very good bug tracking system called FogBugz and they offered a product called Kiln which provides hosted hg repositories (and good integration with the source control and bug tracking system). I have really begun to hate subversion which is what I have been using the last few years. One of the main annoying things about SVN is issues about copying folders or projects. Eclipse is not smart enough to *not* copy the .svn folders so you quickly end up with a mess if you forget to delete them in the destination. And there are other problems, so I was hoping that hg would be better.

It took tens of hours and many tries to convert my 130 or so projects containing about 1.5G of source and binaries to hg from SVN. This was time mainly spent pushing the newly created repositories up to the Kiln server. Sometimes it would run for several hours and die, or timeout, giving no indication as to what was wrong. And hg push does not seem to push one changeset at a time, unless you tell it to; it’s pretty much all or nothing. So if you are pushing 100 changesets and 90 got completed and it dies, you still get nothing in the target repository. There is also no status indication as to which changesets have been processes. The Kiln folks have provided some useful extensions that make it try to batch the pushes into groups but even this does not work in all situations as the heuristics for guessing the batch size are not good in all cases. Sometimes a single changeset can be huge (I have several big things checked in). I thought this would all get better once the initial work was done.

But that was not the case, I decided to reorganize some large projects by moving them from folders to the top level folder. Pushing to the hosted repo took several hours. And also the Eclipse tooling for hg (version 1.6.0 of Hg/MercurialEclipse) did not seem to figure this out correctly. Even hg seemed to be confused about this (I did an hg mv and then hg commit, but it left behind the old folders).

Another annoying thing was that empty folders from svn were simply dropped in the conversion.

Finally the Eclipse tooling is really not there. When I start up Eclipse with my 130 projects it takes nearly 4 minutes of the CPU being pegged for the hg status to be updated in the project explorer. Compare this with about 10 seconds with svn (yes I have filed a bug about this with the HgEclipse folks — they seem to be very responsive [I have filed other issues which have been fixed quickly]). And it seems to be very slow in responding to changes like ignoring certain files. And this was only after a few minutes of using this with all of my projects. It seems to want to refresh resources quite a lot which is not workable with my amount of code.

And Kiln itself had troubles. I would try to open certain directories using their web support and got a cute message about the Kiln being overheated and they were working on it.

I’m sure all of these issues will be worked out over time, but I have given up after a week of fighting this stuff and am going back to svn. I look forward to trying in a few months when hopefully these issues will be resolved. I am using version 1.5.2 of hg on an Ubuntu system.

CNF Testing for 3.5.2 – Please test your projects

If you are a project that uses the Common Navigator/Project Explorer, please be sure and test over the next week with the 3.5.2 RC2 build.  A few critical fixes have gone into the CNF and we need to know sooner than later if there is a problem:

296253	maj	P3	Linu	francisu@ieee.org	RESO	FIXE	[CommonNavigator] An empty label is not properly shown when it is the only contributed label
295803	maj	P3	Linu	francisu@ieee.org	RESO	FIXE	[CommonNavigator] Source of Contribution set to lowest priority NCE, not the NCE providing the children
296728	nor	P3	Wind	francisu@ieee.org	ASSI		[CommonNavigator] Problem with enablement on navigatorContent extension point
299438	nor	P3	Linu	francisu@ieee.org	RESO	FIXE	[CommonNavigator] CNF viewer state non properly reset when NCEs are activated or deactivated