Sadly, converting from Subversion to Mercurial *almost* failed (revised)

(Since posting this I’m more encouraged and I am keeping going, in fact I have now committed to the switch — there are some serious performance problems for my large repo, but given the responsiveness of the HgEclipse folks I expect it will be resolved soon, and I am able to get work done — Thanks Ekke for your comments and encouragement about this)

I really loved the idea of using a DVCS and I settled on Mercurial (hg) because I use a very good bug tracking system called FogBugz and they offered a product called Kiln which provides hosted hg repositories (and good integration with the source control and bug tracking system). I have really begun to hate subversion which is what I have been using the last few years. One of the main annoying things about SVN is issues about copying folders or projects. Eclipse is not smart enough to *not* copy the .svn folders so you quickly end up with a mess if you forget to delete them in the destination. And there are other problems, so I was hoping that hg would be better.

It took tens of hours and many tries to convert my 130 or so projects containing about 1.5G of source and binaries to hg from SVN. This was time mainly spent pushing the newly created repositories up to the Kiln server. Sometimes it would run for several hours and die, or timeout, giving no indication as to what was wrong. And hg push does not seem to push one changeset at a time, unless you tell it to; it’s pretty much all or nothing. So if you are pushing 100 changesets and 90 got completed and it dies, you still get nothing in the target repository. There is also no status indication as to which changesets have been processes. The Kiln folks have provided some useful extensions that make it try to batch the pushes into groups but even this does not work in all situations as the heuristics for guessing the batch size are not good in all cases. Sometimes a single changeset can be huge (I have several big things checked in). I thought this would all get better once the initial work was done.

But that was not the case, I decided to reorganize some large projects by moving them from folders to the top level folder. Pushing to the hosted repo took several hours. And also the Eclipse tooling for hg (version 1.6.0 of Hg/MercurialEclipse) did not seem to figure this out correctly. Even hg seemed to be confused about this (I did an hg mv and then hg commit, but it left behind the old folders).

Another annoying thing was that empty folders from svn were simply dropped in the conversion.

Finally the Eclipse tooling is really not there. When I start up Eclipse with my 130 projects it takes nearly 4 minutes of the CPU being pegged for the hg status to be updated in the project explorer. Compare this with about 10 seconds with svn (yes I have filed a bug about this with the HgEclipse folks — they seem to be very responsive [I have filed other issues which have been fixed quickly]). And it seems to be very slow in responding to changes like ignoring certain files. And this was only after a few minutes of using this with all of my projects. It seems to want to refresh resources quite a lot which is not workable with my amount of code.

And Kiln itself had troubles. I would try to open certain directories using their web support and got a cute message about the Kiln being overheated and they were working on it.

I’m sure all of these issues will be worked out over time, but I have given up after a week of fighting this stuff and am going back to svn. I look forward to trying in a few months when hopefully these issues will be resolved. I am using version 1.5.2 of hg on an Ubuntu system.

About these ads

About francisu

Senior Architect at Talend, responsible for data transformation and other cross-product issues. Also committer in Eclipse Platform UI (mainly Common Navigator) and maintainer of Jenkins EC2 plugin.
This entry was posted in DVCS. Bookmark the permalink.

8 Responses to Sadly, converting from Subversion to Mercurial *almost* failed (revised)

  1. ekkehard says:

    I never had problems pushing to mercurial – did this for repositories at
    - sourceforge
    - bitbucket
    - and also EclipseLabs
    2 repositories each of around 200 changesets – it took only minutes to push it all
    so I think this problem is perhaps Kiln and not Mercurial
    Mercurial uses transactions to be safe and under eclipse preferences you can set timeouts

    the other problem of refreshing multi-project-repositories is sometimes frustrating:
    * MercurialEclipse needs 50 seconds to refresh my 100 redview projects from 2 repositories after startup of eclipse
    * redraw of decorations sometimes needs 10 seconds

    reported this at HgEclipse – and as you wrote: in most cases they fix problems very soon
    ekke

  2. ekkehard says:

    did some more tests:

    first commit after startup needs 15 seconds

    following commits only need 3- 5 seconds,
    doesn’t matter if all projects are involved or only one single

    sometimes many projects appear for very short time in progress view,
    sometime only one project

    where commit itself was fast – decorations always took around 10 seconds,
    so the visual feeling is that commits are slow

    pushing to sourceforge took 9 seconds
    pushing to bitbucket took 40 seconds
    pushing to eclipselabs (google.code) took 5 seconds

    …think this all runs fast enough ;-)

    only problem is the long start-up-time and delay until decoration was refreshed

    ekke

  3. Francis Upton says:

    Thanks for your comments Ekke.

    Some of my projects are very large, and in total it’s around 1.5GB that seems to get refreshed at startup that’s why it’s taking so long (4 mins compared to your 50 seconds), and it also takes a huge amount of memory (I run with the heap status). When moving things around I have gotten out of memory errors even though I run with Xmx1200m.

    I think the pushing also might be related to the size of my stuff, for example I have a few of entire Mule ESB checked in (including all of the Jar files) as one of my projects, and then if I move these around, it seems to have to copy all of the content during the push. My upload speed is measured at about 90KB/s which is pretty much the wire speed for my connection, so I don’t think the problem is Kiln-specific.

    It would be much better when pushing if there was a way see progress with these large pushes, both seeing each changeset and when there is a large changeset to see the progress of that (how many MBs pushed and to push).

    If the initial startup refresh time and memory problems can be fixed soon, then I may immediately reconsider going to hg (unfortunately I would have to reconvert everything from svn again, but know that I know most of the tricks this should go faster). But I remain concerned that MercurialEclipse is not as stable as the SVN/CVS team support and that I will see other problems that block me. That being said, I’m normally happy to be an early adopter so I’m willing to put up with issues.

    I am thinking that my hg mv issues were beyond the scope of what MercurialEclipse was expected to deal with. I did hg mv to move my projects in my local repository and expected that my imported projects would see this, but they did not, so I think the correct answer was to delete them and reimport them.

    One of the other issues I have is that I’m new to hg and therefore don’t know what to expect in some cases. For example when I said yes to “delete contents on disk” after I had cloned a repository in creating the projects, HgEclipse proceeded to permanently delete the projects from the repository which was not at all what I was expecting (though I can see how it made sense from an hg perspective, it’s not the way that SVN/CVS works).

  4. Francis Upton says:

    I think the bulk of my problems are because of the size of one of my repos (the 1.5GB one). For example adding a folder takes over 1 min, regadless of the size of the project.

    http://www.javaforge.com/issue/11930

    I really want to make this work, so I’m keeping going.

  5. ekkehard says:

    …have seen your issues, hopefully the hg team can make some parts faster.
    from my experiences last months they’re doing a great job and enhanced hg to make working with multi-project-repos better and easier

    …getting more informations while pushing as mentioned above sounds as a good idea – perhaps you can add a requirement

    …if you delete projects with delete content from disc, then hg should come up with a commit dialog. now you can decide: commit the delete or cancel the commit, then the project should remain into the repo

    ekke

  6. Francis Upton says:

    After some exchanges with Alexi and testing more it seems clear that that my problems are related to the number of projects and size of my main repo. The repo contains 114 projects, a total of 3059 folders and about 98000 files. One project is about 500M, two are about
    250M and the rest are pretty small, about half of the projects are only a few K (Eclipse feature projects). About 125
    of the files exceed 2MB, 4 exceed 5MB and 1 exceeds 10MB.

    So at the moment, it’s very painful (essentially barely usable, depending on your standards) to use HgEclipse with a repo this size; any operation is very slow (minutes) to refresh decorators so it’s hard to tell where you are until things settle down. However I have every confidence these issues will be resolved soon. Here is a link to the bug about this:

    http://www.javaforge.com/issue/11928

  7. Chris Aniszczyk says:

    I’ll personally convert you to Git in a couple months once the tools are ready.

  8. Lars Vogel says:

    Actually fun reading PlanetEclipse after a period of vacation. I first read your blog post about the solution and now about the problem. Thanks for pushing Mercurial with your large projects, I guess the EGit project will also be happy for an large example. :-)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s