Amazon EC2 plugin – 1.30 Released

A significant bug fix release was made in the plugin, thanks to all to helped test it and provided contributions.

The next release will probably be in the next month or so and will include the acceptance of enhancement PRs that have been filed.

Version 1.30 (Jan 23, 2016)

  • Add config to prefer the public IP to private IP when ssh-ing into slave
  • Added common method to compute tag value and also created constants for demand and spot
  • JENKINS-27601 instance caps incorrectly calculated
  • JENKINS-23787 EC2-plugin not spooling up stopped nodes
  • Depend on the aws-java-sdk plugin to limit AWS SDK duplication
  • Upgrade AWS SDK to 1.10.26
  • Terminate instance even if ec2 node deletion failed
  • JENKINS-27260 SPNEGO for Windows in the EC2 Plugin
  • JENKINS-26493 Use new EC2 API endpoint hostnames
  • JCIFS first tries to resolve a dfs path would timeout causing a long startup delay
  • JENKINS-28754 Jenkins EC2 Plugin should show timestamp in slave logs
  • JENKINS-30284 EC2 plugin too aggressive in timing in contacting new AWS instance over SSH
  • Use AWS4SignerType instead of QueryStringSignerType
  • Add minimum timeout for windows launching
  • Better exception handling in uptime check
  • JENKINS-29851 Global instance cap not calculated for spot instances
  • JENKINS-32439 JENKINS-32439 Incorrect slave template (AMI) found when launching slave
  • Improve logging to be less verbose
Posted in Jenkins EC2 Plugin | Leave a comment

Amazon EC2 Plugin Release 1.30

I have just finished a bunch of work on this plugin to fix a number of bugs related to starting instances and respecting instance caps for spot instances. The plugin will now restart a stopped instance and always respect the instance caps. Other issues about the credentials working correctly in all regions should be fixed.

Before I make this release, I would like to have people test with the 1.30 Snapshot to see if there are any problems. Once this release is done, I will take some new features and move to require a more recent version of Jenkins.

Posted in Jenkins EC2 Plugin | 1 Comment

Moving to a Mac

With my new gig at Talend I have requested a MacBook Pro as a company computer, thinking that since I’m now an architect I will have to do more writing and (god forbid) slides. Of course I will continue to develop software, both for the company and in my open source work. For years, my preferred development platform has been Unix (shows how old I am, I really mean Unix) or Linux. And I have had to work on all three of Windows, Linux and the Mac to test my software. Though I had a Mac, I did not work on it very much. Thanks to Eclipse things just worked there pretty well, I only had a few Mac-specific issues.

So I started to move everything to the Mac and I was shocked at how good it was. Really just good. Everything just feels better than the Ubuntu GUI that I was using (Gnome 3 — I just could not deal with Unity). I spent the money and got the Thunderbolt display and it’s just amazing, so big and clear, and it is essentially a docking station. Everything works fine with my odd keyboard and mouse.

Unfortunately, there is an Eclipse SWT issue (it looks like) which is preventing me from running all 3000 or so of the data transformation unit tests on the Mac, so I have spent a couple of days working on characterizing it, and the thing that I found really shocking about the Mac was Instruments. I could get a ton of profiling information from a running program with an amazing GUI with no effort. By combining this with JProfiler, I should be able to get what I need to characterize (and even possibly fix) this leak problem.

I have never considered having Windows as my daily machine, and now that I have had the Mac, I’m never going back. It’s just that good.

Posted in Talend, Uncategorized | Leave a comment

Now at Talend

After many years of leading Oakland Software to develop really good data transformation technology, I have moved to a new phase, having sold these assets to Talend. A little while ago, I joined Talend as a Senior Architect responsible for data transformation as well as other cross product issues and I’m very excited to be part of this team of extremely bright and talented folks.

For quite a while now the Oakland Data Transformer has worked well with the Talend ESB runtime technologies, and now that it’s part of Talend we will work on integration at the Talend Open Studio level for a future Talend release. You will hear more about that as the work progresses.

I am very grateful to the open source community, mostly to my friends at Eclipse on which my product is based. It would not have been possible to make such a high functionality and robust product without the extensive infrastructure provided by Eclipse to do such things. I think however that the most important part of Eclipse is the culture. It’s a culture of openness, mutual respect, and encouragement. People are encouraged to be nice, helpful, and not at all arrogant and this is the case with everyone I have worked with there over the years. It’s been an amazing experience being part of it and I hope to continue my small contributions to Eclipse.

Posted in Talend | Tagged , , , | Leave a comment

Jenkins EC2 work

About a year ago, I have gotten involved pretty extensively in the Jenkins EC2 plugin where I’m now the maintainer. This was motivated by wanting to move my company’s (Oakland Software — now mostly part of Talend) build process to EC2 and finding out that it was simply not possible unless some significant work was done on the plugin. Seeing that the plugin was missing an active maintainer, I requested the responsibility (which was quickly granted as it the custom in Jenkins).

I have done 3 major releases in the last year with both contributions originated by me and from many members of the community adding significant features like:

  1. Support stopping instead of terminating instances
  2. Allow the use of EC2 spot instances
  3. Proper support for multiple clouds
  4. Greatly increase the accuracy and reliability of starting and stopping instances and adhering to limits
  5. Many bug fixes

I remain active with this project and working closely with several members of the Jenkins community to improve it. With these improvements the plugin has become more popular gaining nearly 200 new installations in the past year.

In addition to coordinating and helping with contributions, I would like to start a conversation about looking the new Google Compute Engine service, which is similar to EC2. We should be able to leverage what we have learned in the EC2 plugin and to the refactoring necessary to cleanly support the Google service in addition to Amazon’s. Ideas and contributions are welcome.

Posted in Jenkins EC2 Plugin | Tagged , | Leave a comment

Some thoughts on the Java u21 problem

I see all of the platform UI bugs and caught my attention and I have made several comments on it. I thought I would share my observations and suggestions at this point, being a member of the Eclipse community.


  1. This problem happens when you run Helios on Windows (possibly other platforms) with the current version of Java (Java 6 update 21). The problem also manifests in the worst possible way, it shows up as a hang (or possibly and infinite loop). As I write this, I’m sure that hundreds of hours are being spent trying to figure out what’s going on by people trying to use Eclipse.
  2. If there is such a thing as an emergency about things not working right, I would say this would be it. Downloading the current stuff is broken.
  3. This is an Eclipse bug. We are using the internals of software that we should not be, it changed under us and it broke our stuff. Also we did not certify with this software. We need to fix this problem. Complaining about that Oracle changed it and depending on them to fix it is a disservice to our users. There are many many examples of this sort of thing happening (mainly in SWT) and in most cases Eclipse seems to do the right thing by working around the problem and fixing Eclipse so that the people using Eclipse have the best experience possible.
  4. The bug happens with many previous versions of Eclipse due to the Oracle VM change, and some solution needs to be provided to them.
  1. Fix Helios right now. Today if possible. Yes this will require a 3.6a or whatever, but we have to make this configuration seamlessly work.
  2. I know a lot of work has been done to develop workarounds and solutions for the previous releases. These need to be widely communicated, including on the front page of We need to get the word out to help people. And we have to keep in ming people will be looking for a solution to a hang.
  3. Improve our release certification process. There is a Java release process where they test and and provided releases available for testing and certification precisely so that these problems don’t happen. We are one of the leading IDEs for Java, we should be totally on top of this process (including having relationships internally with Oracle so that mechanism are in place such that stuff does not get broken if necessary). We have the means to see this is coming and we should have prepared for it by certifying Helios on the u21 version of Java. I know we are pretty formal about talking about our reference platforms, we need to do better here.
  4. Have some policies about accessing the internals of dependent products. Of course it’s necessary to do this in a lot of areas (particularly SWT and Equinox), but where this is done there should be some extra review and also some more centralized tracking so that we understand these dependencies and can consider this in our certification. Also having the extra review will minimize the temptation to quickly add something that’s an internal dependency. People should try to think long and hard about this before doing it to see if there is another way.
This problem is a very serious problem, and we should do like what they do when they investigate airplane crashes, we should look at this from every angle and get as much learning  out of this as possible and implement improvements of the process at all levels to make sure it does not happen again.
Posted in Uncategorized | 4 Comments

MercurialEclipse and Andrei the “handle it” dude

As I posted previously I was having real trouble with hg and MercurialEclipse.

It turns out a lot of my problems with slowness in decorator updating and startup with Eclipse were related to the number of projects (and maybe size) of my repository. The repo contains 114 projects, a total of 3059 folders and about 98000 files. One project is about 500M, two are about 250M and the rest are pretty small, about half of the projects are only a few K (Eclipse feature projects). About 125 of the files exceed 2MB, 4 exceed 5MB and 1 exceeds 10MB.

I reported a bug on this yesterday and immediately Ekke (an hg/MercurialEclipse fan) made some tests and posted them there, Andrei (the MercurialEclipse maintainer) asked several questions and now, less than 48 hours after reporting the bug report, I have received two updates of MercurialEclipse, the second of which reduced the start of time from close to 4 minute to 30 seconds, which is in the acceptable realm.

A while ago I had a similar response from Andrei with a showstopper issue for me which was resolved in less than 24 hours.

I am very happy I made the decision to go with hg, even though it’s a relatively new path from a number of perspectives and the main reason I’m happy about it is that I know that any real problems will be taken care of immediately, and that’s something that’s very rare in this business, no matter how much money you are willing to pay.

Thanks Andrei!

Posted in DVCS | 1 Comment