This blog will guide you through the thought process and some implementation details of how you would go about developing a simple application using PHP and MySQL and then show you how you could re-architect the system using WSO2 products. How and why they’re used. It’s written in a very high level way so that you can follow along easily. Objective of this blog is to introduce you to a familiar topic, developing a simple web application and then when different requirements spin up possible problems you have to consider and how WSO2 products provide rich and easy way of solving those problems.
When you’re new to WSO2 platform it can be a bit overwhelming and you might feel that you’re at a loss. Specially if you’re new to web services and haven’t worked with similar software systems before. When you’re starting out you might wonder where to start as the same task can be done in different ways and using different products. Choosing the right products to use for your situation can also be challenging. Fortunately it’s not all rocket science, get use to the lingo and understand a few basic things and you’ll be on your way in no time!
Before diving into the list of products and what they can do, let’s see some points about application design. If you’re a web developer designing websites for clients you’re probably familiar with blogging tools like Wordpress/Drupal/Joomla etc… If it’s a relatively simple website all you have to do is create a custom theme, putting the right logos/colors and layout and you’re done. You’ll be spending most of your time on the frontend aspects and allowing the underlying content management system you’re using, to update/add new content to the site easily.
Then there are systems that you don’t want to use a content management system. Those situations where using a CMS complicate things than it helps. Simple web applications. For those, you would need a simple server side programming language like PHP and some HTML/CSS to shape up the frontend. Most web applications start this way. You start with a bunch of PHP pages and then when there’s a need to talk to the database you write a “light weight” DB layer which sits on a class of it’s own, you instantiate that class and carry on with the DB operations. Then there’s a whole range of patterns in this area that documents best practices when it comes to developing web apps. MVC, FCP and others. Then there are frameworks which capture these best practices and try to provide you with high level abstractions. CodeIgniter, CakePHP, ZendFramework comes to mind.
So if you’ve been working with this space for some time, web services might not make sense to you. Why do you need web services at all? For example why would you need something that expose data in a database as a web service? That seems ridiculous when you can query the database using mysqli_query() right?
Why do I need web services? I can develop a far more simple web application
with PHP and MySQL!
You’ll probably will go through or might have to come up with an architecture for your web application similar to web services when you have to think about scaling. When that happens you need to be able to treat different parts of the system independently and being able to concentrate on certain functional aspects of the app without having an adverse impact on other parts. You need an application design that distribute functionality into different components. Even though this is a gross over simplification bare with me for a second. Then you need some way of communicating with these distributed components of the system. Using web services is one such solution to the problem and no the only one.
Let’s see how requirements spin up in a typical project that you might have to deal with. First it might be a request like, we need to have an internal web application that give a nice user experience to all our functions in the bank. If you’re tasked with this, then you would need to dive in to what are these functions in the bank and then come up with a web application that will have all these functions. In the beginning it might be such that you’re given a single module as a pilot to see whether it’s successful or not. May be with the savings accounts module in the bank. Let’s say you dive into the requirements, use a DB for transactions and code up the site using PHP. Nice, simple and functional. Have a nice front end everyone is happy and you’re given the go ahead to include more functions. Like, fixed deposit module, credit card processing module, loans processing module and so on. For this example even though I took a bank it still applies to wide variety of systems and companies.
Thinking about possible application extensions, big picture of how your
application will get used in the long run, how it will play along with
the stuff you already have in your organization will save quite a lot of
time when you’re faced with integration tasks down the line
Developing a simple app
So, continuing, if you’ve thought about having all these modules initially then you’re job is easier. If not you need to refactor the code and break it into modules. Cool. You code everything up, hook up the DB, have sections in the UI for all the modules and everything is progessing nicely. May be you put all different modules into a structure like,
modules/ common/ fixed_deposit/ loans/ savings/ ...
And all the functions that are common to all the modules put into the commons folder. Standard stuff. Easy. Now, if you’re really lucky, initial requirement doc lists requirements to monitor the whole system. If you’re unlucky monitoring requirement will come up when you’re done with the system. You need to somehow log all the operations that are happening in the system and provide a monitoring UI for admin users to find out how the system is progressing, data usage, what modules are getting utilized and what are under utilized and so on. This becomes not only an operational level issue but a management issue too. For example if fixed deposit rate is going down may be it might help to do some marketing and give lucrative deals to attract more fixed deposits. You haven’t thought about monitoring, now you have to add code to log these information in your modules. Then create a dashboard. May be you can create a separate tab in your UI that’s visible only to admin users and selected roles in the management to give a view for this data. Again, not that hard. Your application will look something like the following.
At this point if all people were happy and set for the next 10 years then it will be terribly boring. Also it rarely works that way in real life. When you put the system into operation all kinds of requirements can come up. Let’s see some of those,
- I need to be able to have the monitoring UI on my iPad, all the operations people should have this iPad/iPhone app to see how the system is performing
- We need to have user editable set of rules that allow business users to edit/modify logic of the rules. We don’t and wont edit PHP code everytime when the interest rate is reduced
- We need the fixed deposit information to be integrated to the new application we’re building that’s outsourced
- We need to expose a subset of information about loans and customers to other banks for creating a black list of bad customers
- We need a way to quickly detect fraudulant transactions from our credit card transactions
- This system is great, we need to integrate it into the other old systems that we have in our bank. We have some old systems that we need to pull data and display on the new website
- We’re going to use a new cheque processing system from another bank, for that we need to send all our check related information to that system securely over the internet. We have to make sure we send this information in a reliable fashion
If you think ahead, since the system you developed is a single PHP application that utilize a database, there are other concerns as well. If for example, savings account module is heavily used and generate a lot of traffic in the system, this is not only affecting that module but the entire system altogether. If the entire system is down then that’s a big impact on the business. That’s certainly not what you want. PHP application we consider is a monolithic one. Meaning everything is in a single app. Breaking this down to several functions is easier said than done. One option is to break this down and use XML-RPC to do remote procedure calls. When you break functionality down to several pieces and use a bunch of machines to host there are additional concerns that pops up. Like for example, security. You cannot talk between modules with plain text messages because information that you will be sending back and forth are sensitive. It’ll be related to customer information, accounts, credit card information and so on. So you have to at least host the functions that you’ll be hosting other machines through HTTPS. If you need data encription even though it can be implemented, you have to implement in all the different components having a large number of code duplication. You have to maintain and be concious about all this code and test each and every component for functional accuracy with all the additional stuff. Then you have to rethink about monitoring aspects. Same goes for the set of business rules as well. OK, now you have everything separated, then what about that savings accounts module which attracted a lot of traffic? How would you go about scaling that? You can have a bunch of Apache servers having hosting the savings account module and have a load balancer load balance among those. If you have used PHP sessions, and if your functions rely on state that’s stored in the current session, now either you have to enable session replication among the nodes or re-factor your code to make them stateless. Which is going to require some rethinking of doing things. Then, when you have all these changes, you have to get it tested properly for inconsistencies and functional accuracy. It’s a lot of things to consider for a simple PHP application.
Now you understand the problem and complications that all the additional requirements/factors will introduce now let’s look at the WSO2 platform and see how those can help in this situation. Usually when people think about web services in general they think of heavy weight enterprisy stuff that brings unusual complexity and baggage with them that it’s so difficult to manage in a meaningful way. That’s why most of the people prefer to go with simple languages like PHP and have a very lean setup. Another fact is people have heard horror stories of many SOA project failures. People who got bruised by those experiences will not touch anything related to SOA with a ten foot pole. But they will be wrong. Very wrong as we’ll soon see. Ofcourse, if you look at products from, say Oracle, related to SOA they ARE in fact very complex. Multi gigabyte downloads alone should send you a message about what to expect. And you cannot expect anything more from a company who’s run by lawyers, not software engineers.
Choosing the right technology
When you’re choosing a technology for implementing a project, traditionally it boils down into few choices. Leveraging what you already have in place. Like for example if you have an application server, using that. Then ofcourse choosing a technology that’s familiar to your development team is another key. Today however, these choices are changing. What technology gives you the highest turnaround time when it comes to implementing a project seems to be dominant in many situations. Also you have to manage flexibility and maintainability aspects. Ability to adopt to new requirements easily is very important. When you have a new module to implement (still talking in terms of our bank example) if it takes 6 months to get everything going it’s not going to be effective.
Let’s recap our new requirements and think how we’ll effectively re-architect the system.
- Your application now should support multiple UIs on multiple devices.
- Have to abstract out some business logic into a set of rules
- Information should be easy to integrate into other systems
- Information should be easy to securely expose into different systems and people. May be role based access to information.
- Should be able to identify patterns from the transactions that’s performed
- Should be able to extend to adopt and integrate into legacy systems
- Have to be able to expose and send information in a reliable fashion without having any message loss in the system
- Any part of the system should be able to scale easily
- Have to have a monitoring system that’s easy to configure for new modules as well as monitoring the sort of distributed setup we’re going to have
- When we break into functionality and have a distributed setup we need to be able to secure the communication that’s happening between modules and external systems
How WSO2 products fit in
So you have a frontend, also you can allow people to create custom dashboards easily if you want. What now? Now you need to have a set of APIs that implement business functionality. When you have that it’s easy to connect different applications/frontends/apps to get data. If you’re connecting mobile devices then you might want to handle JSON data for ease of processing. Since these APIs can be exposed not only to internal applications but also to external apps then you need to have security. Also monitoring capabilities to monitor who’s using what API, frequency and response times. For this you can use WSO2 API Manager which makes creation and manage API lifecycle through an easily administered web console.
The APIs need to talk to a service or a set of services which implement business functionalities. Since we’re dealing with modules like fixed deposit, loans, cheques each of those can be implemented as a separate service and group all related functionalities to that service. Web services encourage you to do everything in a stateless manner so it’s easy to scale when it comes to that. Service implementation can be done in Java and host it in the Application Server. Application Server is running on an embedded Tomcat instance. Now you have business logic implemented and hosted in the Application Server. Next step we can use the Enterprise Service Bus to make communication between components streamlined and act as a point that integrate different systems. Legacy systems in our example.
Then you can develop a data services layer on top of the database that you were using to exposed those data as web services. So, when you implement business services, those can talk to data services for doing data level operations. This provide a clean interface for interacting with database and make application logic much simpler. You will not need to use JDBC calls or use a separate persistence/ORM layer like Hibernate. When you’re accessing data in the database all you have to do is make another web service call. Then data access layer can have it’s own mechanism of security independent of other systems. This is easy with the Data Services Server where you can expose any database table or a custom query as a web service.
Now comes business rules. Business Rules Server can be used to create a set of reusable rules from which your business logic can then refer to. BRS will expose all your business rules as a web service. So, when you’re calling a business rule to evaluate, all you have to do is call another web service. Let’s see how this solution looks like,
As you can see in the picture we’ve included WSO2 Business Activity Monitor (BAM) for monitoring. In API Manager, ESB, AppServer etc… all have publishing agents that push statistical data about service invocations to BAM. You only have to give connection information and details about service statistics will be collected on BAM. Then you can configure UES to display a custom dashboard of all or subset of events. One good thing about this solution is when it comes to scaling you can scale each and every component without having an adverse effect on other parts of the system. If savings account service is attracting a lot of traffic then you can cluster the Application Server. If your system is generating a lot of events that are sent to BAM then you can cluster BAM and so on.
In the above diagram there is a separate ESB which deals with integration aspects. Also it’s there to provide and connect with legacy systems that will be used.
Out of the requirements that we talked about there’s one more bit that we haven’t considered in the picture above. That’s related to identifying patterns about fradulent credit card transactions. For this we can use WSO2 Complex Event Processing Server (CEP). Since CEP analyses event streams natural place to put CEP is with BAM itself. Because we’re sending all events to BAM anyway for monitoring purposes. Then you can configure different alerts based on the criteria you need.
In summary we looked at a simple web application and with requirements come for integrating additional systems and scaling different parts, possible complexities that it introduce to make things complex. Then we looked at if you think of these problems and anticipate some, how you can have a different architecture for the program you’re designing. Also how some of WSO2 products fit into that problem space and solve some of the tricky issues without much effort.