SOA Governance Scorecard

I stumbled on an unanswered question how can we evaluate a governance solution on all the factors given by you? for the SOA Governance Evaluation Guidelines article on my old blog that somehow survived my separation from Sun few years ago and its subsequent swallowing by Oracle. In fact, I have been getting a lot of inquiries lately about the stuff described in that blog. And, although I have not been focusing on SOA Governance and Security for the past two years, every time I do cursory research to answer a question I get the impression that things have not changed much since I left that scene.

So, in case Amit still needs the answer to his question, I have dug out the SOA Governance Scorecard which I have defined five years ago to measure and track us against the competition. It uses 62 evaluation criteria organized in three-level taxonomy. Quite a few people have found it useful and a year later Accenture even adopted it almost verbatim as their Accenture SOA Governance Vendor Analysis v 1.0. As completed my archeological excavations, extracted the scorecard from the dig and gave it a gentle cleaning my trusted brush, the emerged criteria seemed surprisingly relevant: the only things I would have changed if I were writing it today would be expand the aspect list in section 2.1 and added a new top-level section on Operational Readiness along the lines of requirements outlined my article on Brownfield-friendly Governance.

A sample scorecard based on this criteria can be downloaded here.

Scorecard Criteria

1. Governance

overall weighted rating for the entire Governance Section.

1.1. Defines a comprehensive SOA service model

overall weighted rating for the Governance Model used by the solution.

1.1.1. Service Metadata

does the solution have the capability to associate metadata with individual services?

1.1.2. Strongly typed model

does the solution define a comprehensive SOA service model which contains information required by all stakeholders and actors in the SOA landscape? Such service model would be radically different form the metadata-based approaches to SOA Governance adopted by the majority of existing solutions: it should define a strongly typed, easily validated view of the governance domain with enforced referential integrity, while the latter simply adorns the service WSDL.

1.1.3. Model validation

does the solution have the specific capability to validate the model and most importantly the conformance of services and other artifacts? Also takes into the account whether the model only allows after-the-fact validation of published services, or through referential integrity, will prevent users from creating invalid entries.

1.1.4. Enforceable Model

does the solution have the specific capability to enforce the governance information captured by the model during all parts of the governance cycle?

1.1.5. Extendable Model

does the solution offer the capability to extend the model to accommodate customer-specific requirements?

1.1.6. Separate consumer and provider views

does the solution differentiates between the governance data which should be disclosed to the consumers and the data which should only be available to the service providers and infrastructure? For example the fact that certain service requires a valid lease token should be made available to both consumer and enforcement agent, while the policy how to deal with missing and expired lease tokens (whether to fail requests, include warnings, and / or raise alerts) should not be disclosed to the consumers.

1.2. Separates Technical, Business and Governance views of a service

does the solution separate the views of a service from the Technical, Business and Governance domains and allowing them to be expressed and maintained independently? A developer should not be concerned with the cost and subscription model of the services they are developing and consuming, and Business Actors should not worry about the bindings, protocols and lease requirements of the services their organization provides and utilizes. Such separation of concerns and un-concerns should be maintained throughout all aspects of a governance solution starting with the model.

1.3. Separates Service Lifecycle from SDLC

does the solution governance offering maintain clear separation between Service Lifecycle (including service definition, validation, approvals, publication, evolution and retirement) and Software Development Lifecycle of the service implementations (coding, testing, QA, staging, production and maintenance)?

1.4. Breadth of Governance Solution

overall weighted rating for the breadth of Governance functionality addressed by the solution.

1.4.1. Run-time Governance

does the solution support run-time governance functions such as service invocation cycle (find-bind-execute), policy enforcement, security, service virtualization, protocol and binding adaptation, monitoring, management, etc?

1.4.2. Design-time Governance

does the solution support design-time governance functions such as contract, service and service offering definition and lookups, registry-repository access, etc?

1.4.3. Analysis-time Governance

does the solution support service categorization and discovery? Definition and cataloging of reusable governance artifacts such as service and customer tiers, contracts, SLAs, etc.

1.4.4. Lifecycle Governance

does the solution support service lifecycle activities, approval workflows, lease renewal, service evolution and retirement?

1.5. Manages Policies as Reusable Governance Contracts

does the solution support packaging policies into reusable Governance Contracts to prevent uncontrollable policy application and proliferation? Does it help to prevent unnecessary service proliferation by exposing the same service implementation to different constituencies with different contracts?

1.6. Focus of Governance Solution

how focused is the solution on SOA Governance? Higher ratings are given to pure-play solutions focused exclusively on solving the service governance problems, versus other types of solutions such as Management Platforms, ESBs, EAI and Identity Management products and Composite Application Suites that have some SOA Governance capabilities.

2. Support for SOA characteristics

overall weighted rating for the capability of the solution to support various SOA characteristics. Takes into account support for core and additional characteristics, ability of the product to define and support client-specific characteristics and customer's flexibility in choosing which characteristics to implement and how to support them.

2.1. Supports all core SOA characteristics

overall weighted rating for support of the core characteristics.

2.1.1. Security

solution's support for SOA security. Takes into account support for different security aspects: Authentication, Authorization, Integrity, Confidentiality, Accountability, Identity Management and Security Policies; as well as the facets where they manifest themselves: Transport, Message, Application, Asset (Data), Knowledge and Control Security.

2.1.2. Versioning

solution's support for service versioning, including ability to support multiple concurrent versions of the same service, designate primary versions, define version compatibility lists and ability to adapt requests to deprecated versions. Also takes into account to distinguish between version changes that change service interface and those which only affect governance contract.

2.1.3. Lease

ability to support service lease to assure true decoupling of service consumers and producers. Ability to issue, publish and validate lease tokens and implement service lease enforcement policies.

2.1.4. Monitoring

solution's support for service monitoring (both real-time and historic).

2.1.5. Metering and Chargeback

solution's support for collecting service usage information suitable for generating consumer bills. Takes into account support for advanced billing capabilities, such as real-time checks and ability to deny requests for delinquent and overdrawn accounts.

2.1.6. Throttling

ability to define and implement different throttling policies to protect service providers and ensure sustainable service levels. Throttling can include limiting TPS per service, instance, client, etc.

2.2. Supports any additional SOA characteristics

solution's ability to define provision and enforce additional client-specific SOA characteristics.

2.3. Characteristics supported out of the box

number of SOA characteristics which solution supports out of the box (without any additional development).

2.4. Freedom to select any subset of supported characteristics

as the number of SOA characteristics supported by the solution increases, so does potential for overhead. This criterion assesses the ability to support a subset of relevant characteristics during the implementation to ensure that the solution does not introduce any unnecessary overhead.

2.5. Allows incremental rollout of characteristics

solution's ability to change the set of supported and implemented characteristics without affecting any existing service consumers or providers.

3. Architecture

overall weighted rating for architectural maturity and flexibility of the solution and its ability to support all aspects of SOA Governance as the notion of governance continues to evolve.

3.1. Aspect-Based Governance

support for aspect-oriented governance model, which decouples different SOA characteristics and implements them as independent aspect methods.

3.2. Governance Delegation Capability

overall weighted rating for support of the aspect delegation: many of the service governance aspects including: security, versioning, monitoring and management, etc.; have existing packaged solutions broadly adopted in the prospective SOA marketplace. Does the solution allow users who have already adopted a packaged implementation in one or more aspect spaces to delegate the implementation of these aspects to those packages, provided that the packages themselves support such delegation?

3.2.1. Complete Delegation

can the solution delegate entire aspects to an external package?

3.2.2. Partial Delegation

can the solution delegate parts of aspect implementations to an external package?

3.2.3. Orchestrated Delegation

can the solution delegate the entire aspect functionality to and external package but "rearrange" the default functionality through low level APIs?

3.3. Supports SOA beyond Web Services

SOA is often confused with web services, which limits its applicability and adoption. This criterion evaluates how applicable is the solution to other SOA implementation technologies.

3.4. Supports Different Binding Types

overall weighted rating for support of different types of service bindings.

3.4.1. Web Services

support for the standard web service bindings (SOAP over HTTP/HTTPS).

3.4.2. JMS

support for JMS service bindings.

3.4.3. EJB

support for EJB service bindings.

3.4.4. POJO (Java)

support for POJO (pure java) service bindings.

3.5. Customer Defined Binding Types

ability to add new customer-specific binding types, such as .NET, AJAX, COBOL, etc.

3.6. Supports Dual Binding

does the solution support dual (consumer and provider side) bindings? This allows service creation and consumption to occur in multiple environments characterized by different platforms, technology preferences, skill sets, stages of lifecycle, etc. If path to interoperability is seen through the use of single underlying service implementation technology, such as Web Services, it would take unacceptably long time to reach the level of adoption, necessary for SOA to succeed.

3.7. Provides Complete Spectrum of Client Solutions

there is a contention between the need for interoperability and decoupling versus convenience and the need to hide complexities. Can the solution resolve this contention by offering service consumers a spectrum of options for invoking governed services?

3.7.1. Compatible with industry-standard tools

are the services exposed through the solution consumable without any custom client code? Is it compatible with industry-standard tools like Mindreef SOAPScope?

3.7.2. Provides Access points

does the solution provide access points, which allow service invocation without any client-side components, but require client implementation to manage the complexities of dealing with find-bind-execute cycle and compliance with the governance policies?

3.7.3. Provides Client interfaces

does the solution provide client interfaces, which provide service-generic but client platform-specific solution components that insulate consumers from most of the complexities when invoking any exposed service?

3.7.4. Provides Convenience APIs

does the solution provide convenience APIs, which provide both service- and client platform-specific layer for zero-effort invocation of designated services?

3.8. Manages the cost of compliance

does the solution helps to manage the cost of compliance? Is it able to satisfy all of the above criteria would require a solution of significant level of complexity - does it compromise overall viability of the SOA implementations it is designed to support? Specifically:

• Does it introduce performance bottlenecks to underlying service implementations?
• Does it introduce scalability bottlenecks to underlying service implementations?
• Does it introduce a single point of failure or negatively affect the availability of underlying service implementations?
• Does it introduce any additional vulnerabilities or negatively affect the security of underlying service implementations?
• Does it negatively affect the testability of underlying service implementations?

3.9. Service Virtualization

does the solution provide capabilities for building task-specific “virtual” services from existing services? Does it allow to:

• Consolidate one or more operations from different services into one
• Hide selected operations of an existing service
• Rename selected operations of an existing service
• Hide bindings and other details of the service implementation

3.10. Service Refinement

does the solution provide capabilities to slightly change the interface, vocabulary, granularity, semantics or behavior of enterprise services without changing the service implementation or affecting existing service consumers?

3.11. Non-invasive Architecture

is the solution build on a non-invasive architecture, or does it requires to rebuild, re-instrument or redeploy existing service to bring them under the aegis of SOA Governance?

4. SOA Standard Support

overall weighted rating for standard compliance and capability to support SOA related standards. 4.1. Complete support for the Web Services technology stack

overall weighted rating for standard compliance of the existing solution (HTTP, SOAP, WSDL, UDDI / ebXML).

4.1.1. Standard Service Registry

service registry supports UDDI and/or ebXML standards.

4.1.2. Service Repository

solution includes a Service Repository (there is no standard for repositories except ebXML Registry-Repository).

4.2. Capable of supporting all existing SOA-related standards

does the solution has an architecture in place to add support any existing SOA-related standard not currently supported out of the box.

4.3. Capable of supporting any future SOA-related standards

does the solution has an architecture in place to add support any future or custom SOA-related standard not currently supported out of the box.

4.4. Provides hedging against uncertainty in the SOA standards space

overall weighted rating how the solution helps to address and mitigate the uncertainty in the WS Standard space and lack of many needed SOA standards.

4.4.1. Allows to switch between competing standards

does the solution allows to switch implementation of a particular aspect between competing/complimentary standards?

4.4.2. Allows to switch between proprietary and standard aspect implementations

does the solution allows to switch implementation of a particular aspect between proprietary and standard or draft compliant?