We used SoaupUI Pro 3.0 and though there are lot of good things that could be said for it and the price is reasonable it just cost me too many hours due to its shortcomings to recommend it. I haven't actually used any similar tools but paying a bit more for a more reliable and better designed product would easily pay for itself.
A bug in one of the releases prevented us from upgrading past 3.0 (a lot of our test cases would have broken). Despite a bug report being filed, a number of releases have come out without the bug being fixed.
SoapUI has a GUI or nothing paradigm. The test files are in XML but are not formatted for human use (ie really long lines) so it is really difficult to edit the files directly. There were are couple of times when I had to bite the bullet and edit the files directly and it was not pleasant.
There is a nasty bug that stops SoapUI from saving. This bug seems to have been provoked by updating a WSDL in certain ways. Once I figured out which bit was causing the behaviour I had to edit the file to remove steps that were referencing operations that were not the same after the WSDL was changed.
SoapUI is pretty good at adding new tests since the GUI is well designed for that. But making changes to a number of similar tests has to be painstakingly repeated through the GUI. These sorts of repetitive operations would be more easily done by editing the XML but due to the formatting this becomes difficult. I tried reformatting the XML using an editor but SoapUI didn't like the way some of the spaces were transformed so I had to abort that approach.
SoapUI does not seem to allow common sets of steps to be grouped together and called from multiple tests which leads to repeating yourself and means changes later are expensive. Unfortunately in our project we changed our WSDLs a number of times and so got burnt by this a lot.
In summary, I would say that although the bang for buck is pretty good, developer time is not cheap so evaulate more expensive options - there should ne something better out there.
I agree with all of the above, but would also add that SoapuUI doesn't even really support SOAP that well. We had synchronous 2 way, asynch 2 way and asynch 1 way SOAP interfaces. SoapUI didn't really provide good support for verifying any of those scenarios.
ReplyDeleteI'm inclined to disagree and feel that I should put in a good word for the "little engine that could." I've found SoapUI to be a great testing tool for testing web services in a variety of colours - REST, SOAP(http, https and JMS.)
ReplyDeleteThe shortcomings you have encountered, I would suggest, are largely due to the way the tool was utilised. That is, as a UNIT testing tool for a tightly coupled army of "services" of varying complexity written mostly in BPEL. The fact that it could do the job, albeit with a few enhancements, is nothing short of miraculous. No doubt some strange bugs surfaced as a consequence of the rather peculiar way in which you were using SoapUI but I don't think your overall assessment is entirely fair.
In limited use I am inclined to agree with most of what the post says. I've had light of trouble with changes not save, click-only funcaitonality not working, and manual edits to the xml being challenging and unreliable. And I'm on 4.5.2
ReplyDelete