Shifting the Power – Ending Police Traffic Stop Oppression

I sit in my room at the Pact camp. This camp is a significant conference about people (mostly white as it turns out) who adopt people of color. One of the issues that is discussed quite a bit is the issue of race, and in particular, how to handle the problem of black people (mostly men and boys) being harassed or killed by the police. The prevailing wisdom seems to be that you must prepare your black young man, as a boy, to behave in a way that will not attract the attention of the police, and then certainly, if said attention is attracted, to not upset them in any way so that you can be safe. However, this is not really working, as evidenced by the large number of people that continue to be killed by the police.

When this issue of unjustified violence by the police first came to my attention a few years ago, my thought was the problem would be solved once the police had body cameras. I felt that these problems could not exist in an environment where everything was completely public and the police would have to account for their actions in a public way. I did not think much further about it, except for the hopeless feelings I had every I heard about an unjustified death.

I am a computer person, I have always been one. Last week I went to the Hadoop Summit 2016 where we heard several presentations about the promise of big data to dramatically improve the world in areas ranging from game changing medical breakthroughs (cure cancer anyone?), to improving access to education in rural India, to reducing the cost of car insurance. We are only at the beginning of this huge technological shift, brought about the dramatic improvements in accessibility and usability of software, data, and hardware.

I want to see this technology applied to this problem of police oppression, so that we can give oppressed people the tools and power to have safety (the type of safety that I take for granted as a white person) at all times in all places. There are a lot of oppressed people, and there are a lot of people who sympathize with the oppressed people (like me). If we can bring technology to bear on this problem, we can shift the conversation to having the oppressors be the fearful ones (of their actions being exposed), rather than the oppressed. This can also level the playing field, giving people stopped in their cars a way to get instant support.

We have to mobilize and focus the enormous resources of the reasonable people (of all colors) in this country against the very small number of people who think (through action or inaction) that this can be tolerated.

We are taking a number of steps in this direction. There are ACLU apps that allow you to make a video and have it be uploaded so that even if the phone is confiscated the video survives. They also allow you to be aware of bad things happening around you so you can get involved and be a witness. With the use of social networks and improved applications, we can go much further with this.

As I have been writing this, the Philando Castile killing happened. The way this was recorded serves as another example of getting closer to the real time action that’s required: the immediate video of the aftermath was quickly shared on social media. However, what if everything was video and audio recorded from the beginning of the stop? What if Mr. Castile had the opportunity to be speaking with a trained person who could help him as the incident was happening? Maybe this could have been prevented. Even if not prevented, the evidence captured would go a long way in helping justice to be served.

I can imagine a situation where you have an app on your phone, and perhaps a hidden camera or two in the car. When you are stopped for traffic, you just say a phrase into your phone and you are immediately connected with someone who can support you, and everything is recorded both by video and audio. Video recording of police is currently legal in all states, and audio recording of this kind is legal in 47 states. You can let the police know this is all being recorded and that you are in communication with someone in real time. Your location is noted publicly.

I think this sort of setup could be provided at very little (or perhaps no) cost to those who wanted it, and working with the ACLU or similar organization, it’s possible to put together people that can be available around the clock to be the support person. The technology to do this relatively cheaply is there now. It just takes the coordination, will, and some money to make this happen.

If the police knew that everyone who wanted it would not only get a person to help them, but also everything would be automatically recorded, then maybe the police departments would do what was required to fix this. Even though officers might not be adequately prosecuted for this bad behavior, the cost to the policy agency of the unambiguous publicity will be very high and something they would seek to avoid.

Given my responsibilities and qualifications, I’m not in a position to personally lead this effort, or even contribute much in the way of time or money. I’m sure there are better qualified people who can lead and contribute to this. I will do what I can to help put people together and get the word out. Feel free to comment with your ideas of where we can go from here.

Posted in Uncategorized | 3 Comments

Amazon EC2 plugin – 1.30 Released

A significant bug fix release was made in the plugin, thanks to all to helped test it and provided contributions.

The next release will probably be in the next month or so and will include the acceptance of enhancement PRs that have been filed.

Version 1.30 (Jan 23, 2016)

  • Add config to prefer the public IP to private IP when ssh-ing into slave
  • Added common method to compute tag value and also created constants for demand and spot
  • JENKINS-27601 instance caps incorrectly calculated
  • JENKINS-23787 EC2-plugin not spooling up stopped nodes
  • Depend on the aws-java-sdk plugin to limit AWS SDK duplication
  • Upgrade AWS SDK to 1.10.26
  • Terminate instance even if ec2 node deletion failed
  • JENKINS-27260 SPNEGO for Windows in the EC2 Plugin
  • JENKINS-26493 Use new EC2 API endpoint hostnames
  • JCIFS first tries to resolve a dfs path would timeout causing a long startup delay
  • JENKINS-28754 Jenkins EC2 Plugin should show timestamp in slave logs
  • JENKINS-30284 EC2 plugin too aggressive in timing in contacting new AWS instance over SSH
  • Use AWS4SignerType instead of QueryStringSignerType
  • Add minimum timeout for windows launching
  • Better exception handling in uptime check
  • JENKINS-29851 Global instance cap not calculated for spot instances
  • JENKINS-32439 JENKINS-32439 Incorrect slave template (AMI) found when launching slave
  • Improve logging to be less verbose
Posted in Jenkins EC2 Plugin | Leave a comment

Amazon EC2 Plugin Release 1.30

I have just finished a bunch of work on this plugin to fix a number of bugs related to starting instances and respecting instance caps for spot instances. The plugin will now restart a stopped instance and always respect the instance caps. Other issues about the credentials working correctly in all regions should be fixed.

Before I make this release, I would like to have people test with the 1.30 Snapshot to see if there are any problems. Once this release is done, I will take some new features and move to require a more recent version of Jenkins.

Posted in Jenkins EC2 Plugin | 1 Comment

Moving to a Mac

With my new gig at Talend I have requested a MacBook Pro as a company computer, thinking that since I’m now an architect I will have to do more writing and (god forbid) slides. Of course I will continue to develop software, both for the company and in my open source work. For years, my preferred development platform has been Unix (shows how old I am, I really mean Unix) or Linux. And I have had to work on all three of Windows, Linux and the Mac to test my software. Though I had a Mac, I did not work on it very much. Thanks to Eclipse things just worked there pretty well, I only had a few Mac-specific issues.

So I started to move everything to the Mac and I was shocked at how good it was. Really just good. Everything just feels better than the Ubuntu GUI that I was using (Gnome 3 — I just could not deal with Unity). I spent the money and got the Thunderbolt display and it’s just amazing, so big and clear, and it is essentially a docking station. Everything works fine with my odd keyboard and mouse.

Unfortunately, there is an Eclipse SWT issue (it looks like) which is preventing me from running all 3000 or so of the data transformation unit tests on the Mac, so I have spent a couple of days working on characterizing it, and the thing that I found really shocking about the Mac was Instruments. I could get a ton of profiling information from a running program with an amazing GUI with no effort. By combining this with JProfiler, I should be able to get what I need to characterize (and even possibly fix) this leak problem.

I have never considered having Windows as my daily machine, and now that I have had the Mac, I’m never going back. It’s just that good.

Posted in Talend, Uncategorized | Leave a comment

Now at Talend

After many years of leading Oakland Software to develop really good data transformation technology, I have moved to a new phase, having sold these assets to Talend. A little while ago, I joined Talend as a Senior Architect responsible for data transformation as well as other cross product issues and I’m very excited to be part of this team of extremely bright and talented folks.

For quite a while now the Oakland Data Transformer has worked well with the Talend ESB runtime technologies, and now that it’s part of Talend we will work on integration at the Talend Open Studio level for a future Talend release. You will hear more about that as the work progresses.

I am very grateful to the open source community, mostly to my friends at Eclipse on which my product is based. It would not have been possible to make such a high functionality and robust product without the extensive infrastructure provided by Eclipse to do such things. I think however that the most important part of Eclipse is the culture. It’s a culture of openness, mutual respect, and encouragement. People are encouraged to be nice, helpful, and not at all arrogant and this is the case with everyone I have worked with there over the years. It’s been an amazing experience being part of it and I hope to continue my small contributions to Eclipse.

Posted in Talend | Tagged , , , | Leave a comment

Jenkins EC2 work

About a year ago, I have gotten involved pretty extensively in the Jenkins EC2 plugin where I’m now the maintainer. This was motivated by wanting to move my company’s (Oakland Software — now mostly part of Talend) build process to EC2 and finding out that it was simply not possible unless some significant work was done on the plugin. Seeing that the plugin was missing an active maintainer, I requested the responsibility (which was quickly granted as it the custom in Jenkins).

I have done 3 major releases in the last year with both contributions originated by me and from many members of the community adding significant features like:

  1. Support stopping instead of terminating instances
  2. Allow the use of EC2 spot instances
  3. Proper support for multiple clouds
  4. Greatly increase the accuracy and reliability of starting and stopping instances and adhering to limits
  5. Many bug fixes

I remain active with this project and working closely with several members of the Jenkins community to improve it. With these improvements the plugin has become more popular gaining nearly 200 new installations in the past year.

In addition to coordinating and helping with contributions, I would like to start a conversation about looking the new Google Compute Engine service, which is similar to EC2. We should be able to leverage what we have learned in the EC2 plugin and to the refactoring necessary to cleanly support the Google service in addition to Amazon’s. Ideas and contributions are welcome.

Posted in Jenkins EC2 Plugin | Tagged , | Leave a comment

Some thoughts on the Java u21 problem

I see all of the platform UI bugs and https://bugs.eclipse.org/bugs/show_bug.cgi?id=319514 caught my attention and I have made several comments on it. I thought I would share my observations and suggestions at this point, being a member of the Eclipse community.

Observations:

  1. This problem happens when you run Helios on Windows (possibly other platforms) with the current version of Java (Java 6 update 21). The problem also manifests in the worst possible way, it shows up as a hang (or possibly and infinite loop). As I write this, I’m sure that hundreds of hours are being spent trying to figure out what’s going on by people trying to use Eclipse.
  2. If there is such a thing as an emergency about things not working right, I would say this would be it. Downloading the current stuff is broken.
  3. This is an Eclipse bug. We are using the internals of software that we should not be, it changed under us and it broke our stuff. Also we did not certify with this software. We need to fix this problem. Complaining about that Oracle changed it and depending on them to fix it is a disservice to our users. There are many many examples of this sort of thing happening (mainly in SWT) and in most cases Eclipse seems to do the right thing by working around the problem and fixing Eclipse so that the people using Eclipse have the best experience possible.
  4. The bug happens with many previous versions of Eclipse due to the Oracle VM change, and some solution needs to be provided to them.
Suggestions:
  1. Fix Helios right now. Today if possible. Yes this will require a 3.6a or whatever, but we have to make this configuration seamlessly work.
  2. I know a lot of work has been done to develop workarounds and solutions for the previous releases. These need to be widely communicated, including on the front page of eclipse.org. We need to get the word out to help people. And we have to keep in ming people will be looking for a solution to a hang.
  3. Improve our release certification process. There is a Java release process where they test and and provided releases available for testing and certification precisely so that these problems don’t happen. We are one of the leading IDEs for Java, we should be totally on top of this process (including having relationships internally with Oracle so that mechanism are in place such that stuff does not get broken if necessary). We have the means to see this is coming and we should have prepared for it by certifying Helios on the u21 version of Java. I know we are pretty formal about talking about our reference platforms, we need to do better here.
  4. Have some policies about accessing the internals of dependent products. Of course it’s necessary to do this in a lot of areas (particularly SWT and Equinox), but where this is done there should be some extra review and also some more centralized tracking so that we understand these dependencies and can consider this in our certification. Also having the extra review will minimize the temptation to quickly add something that’s an internal dependency. People should try to think long and hard about this before doing it to see if there is another way.
This problem is a very serious problem, and we should do like what they do when they investigate airplane crashes, we should look at this from every angle and get as much learning  out of this as possible and implement improvements of the process at all levels to make sure it does not happen again.
Posted in Uncategorized | 4 Comments

MercurialEclipse and Andrei the “handle it” dude

As I posted previously I was having real trouble with hg and MercurialEclipse.

It turns out a lot of my problems with slowness in decorator updating and startup with Eclipse were related to the number of projects (and maybe size) of my repository. The repo contains 114 projects, a total of 3059 folders and about 98000 files. One project is about 500M, two are about 250M and the rest are pretty small, about half of the projects are only a few K (Eclipse feature projects). About 125 of the files exceed 2MB, 4 exceed 5MB and 1 exceeds 10MB.

I reported a bug on this yesterday http://www.javaforge.com/issue/11928 and immediately Ekke (an hg/MercurialEclipse fan) made some tests and posted them there, Andrei (the MercurialEclipse maintainer) asked several questions and now, less than 48 hours after reporting the bug report, I have received two updates of MercurialEclipse, the second of which reduced the start of time from close to 4 minute to 30 seconds, which is in the acceptable realm.

A while ago I had a similar response from Andrei with a showstopper issue for me which was resolved in less than 24 hours.

I am very happy I made the decision to go with hg, even though it’s a relatively new path from a number of perspectives and the main reason I’m happy about it is that I know that any real problems will be taken care of immediately, and that’s something that’s very rare in this business, no matter how much money you are willing to pay.

Thanks Andrei!

Posted in DVCS | 1 Comment

Sadly, converting from Subversion to Mercurial *almost* failed (revised)

(Since posting this I’m more encouraged and I am keeping going, in fact I have now committed to the switch — there are some serious performance problems for my large repo, but given the responsiveness of the HgEclipse folks I expect it will be resolved soon, and I am able to get work done — Thanks Ekke for your comments and encouragement about this)

I really loved the idea of using a DVCS and I settled on Mercurial (hg) because I use a very good bug tracking system called FogBugz and they offered a product called Kiln which provides hosted hg repositories (and good integration with the source control and bug tracking system). I have really begun to hate subversion which is what I have been using the last few years. One of the main annoying things about SVN is issues about copying folders or projects. Eclipse is not smart enough to *not* copy the .svn folders so you quickly end up with a mess if you forget to delete them in the destination. And there are other problems, so I was hoping that hg would be better.

It took tens of hours and many tries to convert my 130 or so projects containing about 1.5G of source and binaries to hg from SVN. This was time mainly spent pushing the newly created repositories up to the Kiln server. Sometimes it would run for several hours and die, or timeout, giving no indication as to what was wrong. And hg push does not seem to push one changeset at a time, unless you tell it to; it’s pretty much all or nothing. So if you are pushing 100 changesets and 90 got completed and it dies, you still get nothing in the target repository. There is also no status indication as to which changesets have been processes. The Kiln folks have provided some useful extensions that make it try to batch the pushes into groups but even this does not work in all situations as the heuristics for guessing the batch size are not good in all cases. Sometimes a single changeset can be huge (I have several big things checked in). I thought this would all get better once the initial work was done.

But that was not the case, I decided to reorganize some large projects by moving them from folders to the top level folder. Pushing to the hosted repo took several hours. And also the Eclipse tooling for hg (version 1.6.0 of Hg/MercurialEclipse) did not seem to figure this out correctly. Even hg seemed to be confused about this (I did an hg mv and then hg commit, but it left behind the old folders).

Another annoying thing was that empty folders from svn were simply dropped in the conversion.

Finally the Eclipse tooling is really not there. When I start up Eclipse with my 130 projects it takes nearly 4 minutes of the CPU being pegged for the hg status to be updated in the project explorer. Compare this with about 10 seconds with svn (yes I have filed a bug about this with the HgEclipse folks — they seem to be very responsive [I have filed other issues which have been fixed quickly]). And it seems to be very slow in responding to changes like ignoring certain files. And this was only after a few minutes of using this with all of my projects. It seems to want to refresh resources quite a lot which is not workable with my amount of code.

And Kiln itself had troubles. I would try to open certain directories using their web support and got a cute message about the Kiln being overheated and they were working on it.

I’m sure all of these issues will be worked out over time, but I have given up after a week of fighting this stuff and am going back to svn. I look forward to trying in a few months when hopefully these issues will be resolved. I am using version 1.5.2 of hg on an Ubuntu system.

Posted in DVCS | 8 Comments

CNF Testing for 3.5.2 – Please test your projects

If you are a project that uses the Common Navigator/Project Explorer, please be sure and test over the next week with the 3.5.2 RC2 build.  A few critical fixes have gone into the CNF and we need to know sooner than later if there is a problem:

296253 maj P3 Linu francisu@ieee.org RESO FIXE [CommonNavigator] An empty label is not properly shown when it is the only contributed label
295803 maj P3 Linu francisu@ieee.org RESO FIXE [CommonNavigator] Source of Contribution set to lowest priority NCE, not the NCE providing the children
296728 nor P3 Wind francisu@ieee.org ASSI [CommonNavigator] Problem with enablement on navigatorContent extension point
299438 nor P3 Linu francisu@ieee.org RESO FIXE [CommonNavigator] CNF viewer state non properly reset when NCEs are activated or deactivated
Posted in Common Navigator | Leave a comment