Monday, January 24, 2011

BPEL (aka BHell) - is this programming ?

I was fortunate in my 3 years on the SOA project that I had delayed exposure to BPEL.  I managed to get involved in some large components that were done all in Java.  I could hear other developers grumbling about it but realised I needed some exposure to BPEL to know first hand the pros and cons of using it, so I put up my hand to do some BPEL.  Unfortunately, I got what I asked for.

BPEL is a SOA orchestration language that is a dialect of XML.  It is conceived to be coded up by dragging and dropping in a GUI based editor.  Some would conceive it as easy and the future of programming.  My experience of BPEL is sullied by the productivity sapping excuse for an IDE that is called JDeveloper (which is the only GUI based tool that can do BPEL in an Oracle Fusion environment).  As I have commented in an earlier blog post I became so frustrated with this tool that I resorted to coding BPEL in a text editor (Notepad++).   This was also not pleasant as XML is so verbose that the amount of code you can view on one page is limited to what would amount to a few lines in something like Java so you end up doing a lot of scrolling which is frustrating.  You also have to be very careful with namespaces and XML syntax to avoid obscure error messages.

BPEL's deficiencies as a programming language become immediately obvious once you try to do anything non trivial.  You can not define a function - the BPEL SOA service is the lowest level of granularity.  If you want to share functionality between two BPEL services then it needs to be done by defining another BPEL service with associated WSDL, XSD etc.  If you want to have some common code inside your BPEL that is called from multiple code paths in your BPEL then you have to be creative - eg set variables and then do all the common code at the end based on whether those variables have been set OR if you are brave use catch clauses since throwing exceptions will get you to a common point in the BPEL.

Booleans which should be simple are hard to do correctly in BPEL.  A simple if statement becomes a verbose switch/case/sequence monstrosity and a simple loop with a variable count is like going back to assembler.  Namespaces will test your sanity and constructing variables of types that have multiple consecutive optional elements becomes a nightmare of complexity (a default constructed type will have all optional elements populated so you first have to delete them but then the ones you want to add back in have to be explicitly added _after_ another named element.  When you have multiple consecutive optional elements you need to know which of the earlier optional elements are included so you can insert after the correct one).  Lists and maps of course are far too modern to even consider.

We ended up making use of the non-standard Java embedding to maintain sanity when anything other than copying elements from one XML type to another was required.

Similar to what I said with SoapUI the GUI interface is OK for new programs but when you have to modify a number of BPELs in the same way (eg because a WSDL has changed) you are left to slavishly repeat the same GUI steps for each BPEL.

One of my colleagues had a saying that "BPEL makes you dumber".  When it is exhausting just to get something simple done you just can't concieve of doing anything complex.  That can become habit forming.

Usually on a CV you would list as many skills as possible to make yourself look better and get hits on the automated CV searchers used by recruiting people but I would definitely hesitate before added BPEL to my CV because I would prefer to never see another line of BPEL again.

Deadline Driven Delivery - Planning to Fail

In the SOA project I worked on for the last 3 years the management had a peculiar approach which I found difficult to understand and had me concerned about their competence.  I called it Deadline Driven Delivery since it always seemed to be driven by starting with an end date and working backwards.

Three years ago they started with a completely rediculous end date and threw lots of people at it.   The end product was said to be important and stategic to the company so there was never any question of leaving out functionality.  The technology platform (Oracle Fusion) was new and the SOA paradigm was still not mature in its practices within the industry, especially in a performance sensitive environment.  A sensible project manager would be factoring in large contigencies to allow for the many unknown quantities but what can you do when the end date is set ?

Well it doesn't take a genius to work out that we didn't hit the deadline so what happened next ?  That's right a new end date was chosen.  The criteria seemd to be to make it the most aggressive, wishful thinking date possible.  We were told it was going to be "tight and not much room for slippage so we all need to work really hard" - contingency was definitely a dirty word.

Well, wouldn't you know it the real world intervened.  There were bugs, there were holdups waiting for hardware and middleware software updates and a list of Change Requests to boggle the mind.   Then of course there was performance testing that indicated that the heavy weight SOA approach had performance implications.

Each setback seemed to met with the same approach.  Look at original schedule, calculate how much time was lost due to the setback, add that time to the end date of the original schedule,  get approval for the new date, work backwards again.  Hey look the schedule is still "tight and not much room for slippage so we all need to work really hard".

It was clear to me that each project plan was doomed to fail.  If you don't have realistic estimates and you son't allow for contingencies you are just planning to fail.  Unfortunately the bigger the project and more ambitious the project the more contingencies need to be allowed for and the harder it is for anyone to agree up front to fund the project.

If I could see it why couldn't the project management.  Or maybe they could and just realised that was the way the game had to be played in a business setting ?

This was my first taste of the business dynamic in a project.  Before this I worked on projects where the customer would tell us what they wanted, we would prepare a realistic estimate for cost and schedule (with a couple of variations based on number of developers to work on it) and then we might cut functionality or release in stages to meet particular deadlines.  This project had a company board that only met monthly at best, an executive control group, the project management, the development technical leadership and then the rest of us developers.

I was a long way from the action but it seemed as though the project was run on the basis of how much expenditure the board would approve and this was the driver for choosing the end dates.  I am not sure if anyone ever suggested a realistic schedule to the board and got knocked back.  Maybe it is Project Management 101 to never tell the real date for fear of having the project cancelled.  I doubt that anyone would have approved a 4 year project (which it will be by the time it is finally finished) at the beginning so maybe planning to fail is the only way to succeed ?  The project was nearly cancelled a number of times and a couple of the original project managers were sent on their way so it would not be an approach for the faint hearted.

It also appeared to me that no one was really in control of the project.  No one was sufficiently powerful, informed and competent to look at the project, factor in contigencies and approve the real cost to the end of the project.  If such a person existed would the project have gone ahead ?  Hard to say but
perhaps that person would have seen the real folly of the initial schedule and found ways to reduce the size of the project and release it in stages and jettisoned things that didn't have real value (eg reduced use of SOA to only where it made sense).

We would have weekly standup team meetings to look at the schedule (which I came to regard as the "fantasy schedule") and discuss progress.  I tried to determine whether the management actually believed the schedule.  There were over 50 people at these standup meetings and there was question time at the end and I was sorely tempted to ask whether anyone actually believed the schedule or whether they had learnt anything from the 20 failed schedules before but I decided to keep my head down and not bring morale down further.

So were the project managers incompetent, business savvy or just out of their depth.   Still not sure but I am sure that the Deadline Driven Delivery is not a very healthy process to be a part of.

Sunday, January 23, 2011

SoapUI - Can't really recommend it

We used SoaupUI Pro 3.0 and though there are lot of good things that could be said for it and the price is reasonable it just cost me too many hours due to its shortcomings to recommend it.  I haven't actually used any similar tools but paying a bit more for a more reliable and better designed product would easily pay for itself.

A bug in one of the releases prevented us from upgrading past 3.0 (a lot of our test cases would have broken).  Despite a bug report being filed,  a number of releases have come out without the bug being fixed.
   
SoapUI has a GUI or nothing paradigm.  The test files are in XML but are not formatted for human use (ie really long lines) so it is really difficult to edit the files directly.   There were are couple of times when I had to bite the bullet and edit the files directly and it was not pleasant.

There is a nasty bug that stops SoapUI from saving.  This bug seems to have been provoked by updating a WSDL in certain ways.  Once I figured out which bit was causing the behaviour I had to edit the file to remove steps that were referencing operations that were not the same after the WSDL was changed.

SoapUI is pretty good at adding new tests since the GUI is well designed for that.  But making changes to a number of similar tests has to be painstakingly repeated through the GUI.  These sorts of repetitive operations would be more easily done by editing the XML but due to the formatting this becomes difficult.  I tried reformatting the XML using an editor but SoapUI didn't like the way some of the spaces were transformed so I had to abort that approach.

SoapUI does not seem to allow common sets of steps to be grouped together and called from multiple tests which leads to repeating yourself and means changes later are expensive.  Unfortunately in our project we changed our WSDLs a number of times and so got burnt by this a lot.

In summary, I would say that although the bang for buck is pretty good, developer time is not cheap so evaulate more expensive options - there should ne something better out there. 

Tuesday, January 18, 2011

Oracle Fusion - Avoid it if you can

The project I was working on used Oracle Fusion 10g for the last 3 years.  It is scary to look back at how bad it was when we first started using it.  The quality and performance back then were abysmal.  How Oracle gets away with with pushing out Beta quality (if that) and still charges an arm and a leg for it is beyond me.  10g has improved a lot in the last 3 years (it had to) and 11g is supposedly better but I would still stay well away from it if I had any choice.

So why was it selected for the project in the first place ?  From what I can gather the decision was made very early in the project and was no doubt done by managers looking at Oracle sales material.  Oracle software seems to be able to generate a large checklist of inclusions which helps gets the sale and justify the price but once Oracle has signed you up you're stuck.  An extra "bonus" is that they license by the CPU so the worse the software performs the more you have to pay !

Middleware is an interesting product since it costs a lot but doesn't actually do anything by itself.  Oracle's position is that they provide the middleware and then someone else develops on top it to produce the end product.  This situation means the developer finds all the problems and has to deal with Oracle Support to resolve the problems.  However dealing with Oracle Support is such an arduous and time consuming process that one often finds workarounds rather than logging a defect.  A problem will only be dealt with if you can come up with a sample program to demonstrate it and if you have applied all the latest patches etc.  To do this you usually have to figure out exactly what is failing and why.  Our project had a number of people adept at debugging and decompiling the Fusion stack to try to narrow down a problem and understand what a particular Java exception meant.

So what specifically is wrong with it ?
Let's start with the motto.  "Productivity with Choice" is a complete lie.  There is little choice (you're stuck with Fusion - get used to it) and you can say goodbyte to productivity.

Overall quality and user (ie developer) friendliness is a huge issue.  Things can fail in so many different ways, often due to incorrect input but the error messages (if any) don't help work out the problem.  Google usually doesn't help and Metalink's usefulness is limited so that when it finds something helpful it is the exception rather than the rule.  Debugging and decompiling the stack or a lot of trial and error become your tools to diagnose a problem.

JDeveloper looks like a useful tool but it never fails to disappoint, especially with BPEL.  It has a pretty GUI drag and drop designer for BPEL and also provides a source code view into the XML.  Unfortunately the integration of the two views is fatally flawed and you end up losing work.  I am told there is a particular method of saving work before changing views that leads to less frustation but after having garbled text when scrolling (some problem with dual screens I am told), frequently finding changes going missing and then a complete lock up I lost patience all together and started using a text editor (notepad++).  My loathing for BPEL is no doubt related to my frustation with JDeveloper and the suboptimal experience of programming using XML in a text editor but that's the subject of another post.

The ESB caused so many problems that we had to refactor our code to remove it.

The clustering seems half baked.  The way it works is all hidden, there is no API to integrate your own components with it and the Oracle supplied components have varying degrees of support for it.  For example, a BPEL process only needs to be deployed on one node (and then the BPEL engine replicates it to other nodes) but an EJB needs be deployed separately on all nodes.  Configuration changes via the web consoles are not always replicated, etc.

There are plenty of other issues but since I have left the project they are starting to fade (maybe this debrief is starting to work).  Feel free to add more in the comments.

With all the problems you have to ask your self what value is Fusion adding.  If you take the best Open Source Application Server that offers the level of support you need what does Oracle Fusion offer on top of that ?  With an Open Source solution you can avoid the per CPU licensing and you don't have to be stingy with developer systems.  With the money you save you can probably develop any extra components that the Open Source solution might be missing and you can win Karma points by making contributions back to the Open Source project.  You will also no doubt find that your questions and defects get more attention.

So do yourself a favour and evaluate all other possible options before choosing Oracle Fusion.

Monday, January 10, 2011

SOA - What is it good for ?

Well, I won't go as far as the song and say "Absolutely Nothing" but the project I worked on definitely proved that too much of a SOA thing is bad for you.

Somewhere along the line the project management drank the SOA Kool Aid all the way to dregs and mandated SOA everywhere without thought for performance and the impact of having too many services of a low granularity.

We implemented our SOA services using Oracle Fusion (10.1.3.X) middleware as a combination of BPEL processes, EJBs, web services and ESB components.  All SOA services required a WSDL to enable them to be called from other services.  Some WSDLs could be generated from the EJBs but others were pre defined so required JAXB to generate the java objects for interaction with the service.  Some services used WSIF (ie calling an EJB directly rather than going via a SOAP conversion) to improve performace.  

WSDLs and their related XSDs and namespaces are unweildy, verbose and repetitive.  Changing them by hand is time consuming and error prone.  A decent set of high quality robust tools could almost make dealing with them manageable - unfortunately we had Oracle Fusion where the tools were incomplete, buggy and more often than not got in the way.

Testing the services also meant dealing with the WSDLs and the idiosyncracies of the testing tool.  We used SoapUI Pro as our test tool which certainly improved over the years so that it started offering native functionality that we were at first forced to write ourselves using the provided Groovy scripting language.  SoapUI produces XML test files but they were not designed to be changed by humans so when the GUI tool came up short for certain tasks (eg changes affecting multiple tests) it became very painful.

Both Oracle Fusion and SoapUI can be very unforgiving to an error in the WSDL or XSD so that it can be an extensive debugging campaign to determine an error.

Apart from the technology challenges of implementing a service there is the design and documentation side of it.  In theory a Service should have a simple and complete interface, be a well designed, well documented reusable component that can be easily selected from a service catalog.  It should be implemented in a reliable secure environment and enable change management and different versions.  A defect management process is also required since different organisational groups may be managing different services

The above prelude is to demonstrate that there is a fair amount of overhead involved in every Service.  With lots of services you either cut corners (eg documentation, interface completeness) or invest a lot of time and you have to take the performance hit of the service call a lot more often.

In our case most "services" ended up being single purpose and tightly coupled to their dependent services so lost most of the advantages of being a service.  We spent the better part of 2 years modifying and re-modifying our services to achieve better performance and enable more and more production aspects (logging, security, performance metrics) which involved modifying WSDLs and test cases and retesting.  This was very time consuming and inefficient and as a programmer it felt like walking in knee high mud.

Alright so SOA is not good for low granularity service decomposition so what is it good for ?

SOA makes sense for high value reusable components that need to be invoked by different systems.  It is also a good way to make existing functionality (legacy systems) available to other systems in a standardised way by providing a Service Adapter component as an interface to the legacy system.

Ironically that is exactly the way we didn't use it.  There was a legacy system in place that needed to be upgraded.  The decision was made to replace everything in a big bang approach with an unrealistic deadline.  So lots of people implemented lots of services with a new technology stack (when we started Oracle Fusion was lucky to be Beta quality) and focused on functionality.  We ended up creating a big mess that is still being ironed out 3 years later. 

If we had broken the legacy functionality into large granularity services and provided adapters for some of the functionality whilst providing a smaller chunk as rewritten services then we could have focused on getting the smaller chunk production ready and developed our development templates and best practices.  This would have made the project more manageable and given the stakeholders earlier evidence of the project success.  Later parts of the project could then build on the best practices and be more of a known quantity. 

Of course it would have been a harder sell from a project completion date perspective but sometimes you have to be realistic and recognise that when a project gets above a certain size the only way to get success is to be conservative and manage the risk appropriately.

Sunday, January 9, 2011

SOA Debrief Required

I have recently left a fairly large SOA based project and feel like a need a debrief to clear my mind and move on.   Who knows - someone else my read this and find it useful. 

So, I will be adding a few posts over the next few days covering what I think about SOA, BPEL, Oracle Fusion and some of the less desirable management practices that I experienced.

I am trying to keep things anonymous to ensure plausible deniability and allow myself to be reasonably direct so please do the same in the comments.

Of course, these are just the views of one programmer (a nameless one at that) so the usual disclaimers apply.