SaaSkatoon: All Things SaaS!

It’s evident by this point that cloud computing technologies such as Software-as-a-Service (SaaS) and Platform-as-a-Service (PaaS) have changed the way applications are developed.  The interesting thing that we are finding with our customer engagements is that the rapid and iterative nature of designing and developing apps on Force.com has created an entirely new set of challenges with how the apps are tested prior to deployment to production environments.  The ability to demonstrate application features and functionality to project stakeholders in near-real time is more of a double-edged sword than most people realize; on the one hand, being able to show progress and continuously incorporate feedback has fundamentally changed the concept of application development and delivery.  On the other hand, if expectations are not managed properly, the ability to visually represent system designs and demonstrate prototypes in such a rapid timeframe could potentially trivialize the importance of testing, code refactoring and optimization, and change management.

With regards to the testing aspect of Force.com application development and delivery, I have observed some emerging patterns that warrant further study and discussion:

1. Unit testing is real-time and automated.  The Force.com IDE for the Eclipse SDK has been a solid development tool for some time, but the real-time feedback that you get using version 16.0 (Summer ’09) of the IDE with the Eclipse Ganymede release (3.4.x) is of tremendous value.  Granted, the Salesforce API will let you know if there is a problem with your code, but now we’re seeing potential issues called out in Eclipse prior to pushing code to Salesforce.  If you think of unit testing in the traditional sense, where you are testing the validity of snippets of code, the very act of the Salesforce API accepting code that you push to the environment provides immediate validation in an intelligent and automated fashion.  And with the line-item feedback provided by Eclipse, you can catch code validation issues long before saving to the server, significantly reducing the time required to track down and fix invalid code.
2. Integration testing has combined with QA to become a metafunction.  I will be the first to admit that I initially viewed the Apex 75% code coverage requirement imposed by salesforce.com as more of an annoyance than anything.  Now I have come to see test classes not as a necessary evil in deploying apps to production environments, but as a tremendous opportunity to bundle elements of integration testing and end-to-end application QA into a single overarching process.  While most developers dread having to write Apex test classes, we include them in our design specifications as a best practice.  If you think about it, what we are doing is taking the use cases that are fleshed out during the design process and building out test cases from the onset of the development effort; this is a fundamental shift from the traditional role of QA in outdated waterfall-based methodologies that build and execute test cases and QA plans only after development, unit testing, and integration testing are completed.  I have worked with far too many organizations where QA was considered an albatross – the development effort was complete for all intents and purposes, the finish line was in sight, and here comes QA adding weeks, if not months, to the Gantt chart right before that final milestone.  By making QA an iterative process that is tightly integrated with development processes, there are no downside surprises at the end of the development cycle; QA is already done along with integration testing, and it can be argued that the most important aspect of this is that QA is no longer a discreet process – it becomes a living, breathing overlay of the entire development methodology, producing test classes that live alongside the production Apex classes.
3. User Acceptance Testing is a combat sport. What we have found with many Delivered Innovation clients is that if a prototype meets functional requirements, it’s considered “good enough” at that point.  The reality of the situation, 9 times out of 10, is that “good enough” is not quite good enough for a number of reasons.  Namely, demonstrating a prototype in a controlled environment with use cases that are designed to show successful outcomes does not flesh out the true functional capabilities of the app; only when users start beating on an app do we start to see where the stress points are in the functioning of the application.  This is why we encourage what we half-jokingly refer to as “break testing” – we encourage users to “break” the app by any reasonable means.  Realistically, we want this to happen upfront and not two months after production deployment.  Another reason why UAT becomes critically important in the testing process is because the execution of test classes is constrained by governors and limits far more conservative than production governors and limits.  For example, while you can retrieve 10,000 records in a production (non-trigger) SOQL query, you are limited to 500 records when executing tests.  While you may be able to write test classes that execute within governors and limits, once you release the app into the wild you just never know when Salesforce users will find the “unknown unkowns” and use the app in a way that could not have been anticipated…followed by the dreaded ‘Script Exception’ email alert showing up in your Inbox.

How has Force.com, or PaaS in general, impacted your testing and QA processes?