Aspects of PaaS Security

Having a PaaS can yield huge advantages in terms of developer productivity and time to get an application up and running. At the same time it introduce several new security considerations that you should be aware of.

A generic PaaS

If you look at WSO2 PrivatePaaS for example, it’s a generic platform that support pretty much any type of application. Multiple databases (MySQL, Oracle, DB2, PostgreSQL etc…), multiple type of applications Java, PHP, Ruby etc… You can get the full picture of what type of technologies are supported through the above page. This gives immense flexibility from the platform point of view. Also it plays an important role interms of platform maturity.

A generic PaaS which support pretty much anything is a nightmare when it comes to having proper security rules in place. If there’s a standard that says any application using corporate data should talk to existing corp databases on Oracle, then the PaaS should be able to accomodate that. So you should be able to restrict databases exposed to application developers as well. Although there’s a generic PaaS installed it should be able to restrict based on what’s required.

Security spans across what’s provided from the platform to infrasturture level security as well as internal policies.

Infrastructure level security

A PaaS is most frequently stood up on top of an IaaS. This allows the platform to take advantage of cloudiness provided by the infrastructure.

If you’re exposing all middleware services that the platfrom provide for public use, then you don’t need to worry about having any rules at the infrastructure level. Otherwise you need to be concerned about which servers are given access. These requirements change based on the business requirements. Let’s consider a generic deployment diagram first. This outlines the servers that’s there in the solution.

In the above diagram, arrows represent data flows that’s possible between server instances. Users can get to all the applications deployed on the Application Server through a load balancer. From the Application Server node, you can access Data Services, ESB or the Business Workflow servers. You can access the database only through Data Services. There are firewall rules applied at the infrastructure level to restrict access.

Why do you need this?

This encourage adherance to certain service usage patterns. It might look inflexible and to some extent an annoyance. However, for the long run it’s going to encourage best practices. For applications requiring high performance, having to go to the database through another data services layer seems inherently restrictive and bad. From my personal experience this has not been the case. You always have the option of enabling response caching that yield some performance gain. If there’s a really specialized application with special performance requirements then most probably that will be deployed in it’s own set of machines. This you might have read as the “Private Jet mode for tenants”

Retain development time flexibility

The above firewall rules are applied at runtime for end user interactions. How does this translate to a developer developing these applications/data services/integrations? Does it limit or restrict the ability to access these servers because of firewall rules? Does it create more headaches? No!

Worker/Manager clustering to the rescue

WSO2 platform supports a worker manager clusting setup. There are management nodes that’s used to deploy artifacts and there’s worker nodes that serve runtime requests. Let’s look at a diagram,

When you’re deploying, you would interact with the manager node and deploy your artifacts. Manager node will then take the responisibility of announcing to all worker nodes that there’s a new artifact and they’ll all have that within a few seconds. Usually at the deployment time you configure it to route runtime requests only to the worker nodes. So manager node doesn’t serve live traffic. That’s a load balancer configuration. Getting all worker nodes updated with the latest artifacts are done via what we call the deployment synchronizer. If you so choose you can make manager node act as a worker too so it’ll also participate in serving live traffic. Most real world production deployments however tend to keep the manager nodes out of serving live traffic.

In a multi tenant environment, this gives the flexibility of putting certain applications or tenants to their own cluster of machines. Let’s look a bit more complex deployment diagram,

In the above deployment there’s a load balancer per cluster. Also there’s a top level load balancer that forward requests to the correct downstream load balancer. There’s an Application Server cluster. Also there’s a separate Application Server cluster hosting tenant example.com’s applications. PaaS gives you a shared architecture where all tenants’ data are shared across the cluster. If there’s a security requirement that particular tenant’s data should be isolated and be hosted on it’s own set of machines, this gives a cleaner way of doing that. However, it’s still tied into the same platform and services. So even though there are a separate set of machines governing certain applications and data they’re still able to get users/roles/authorization/authentication from the underlying platform. It’s still the same platform, on a private deployment space.

Runtime application security

The next important aspect to PaaS security is isolating data and code at runtime. WSO2 platform is a multi tenant PaaS that has proper isolation between data and runtime code. Even though each tenant is sharing the same JVM and same database instance, each tenant’s data is isolated from the other. One tenant’s users cannot share data between another tenant. You can write an explicit service that deliberately share certain portion of data to other tenants but there are runtime checks in place where it’s not possible to accidentally expose data.

At runtime, Java Security Manager is enabled that prevent access to priviledged operations. File system access, and priviledged methods.

Choice of technology

Even though the PaaS is generic, most organizations want to restrict the type of applications that they allow developers to develop. Also the libraries that can be used within the applications. As a developer, using maven you have the flexibility and freedom of using pretty much any library that’s available. However, this create security issues. These 3rd party libraries might not be thouroughly tested for security vulnerabilities. This can create issues with data loss and security breaches. Usually there should be a library nboarding process before developers are allowed to use a certain library.

Sometimes it helps to reduce choice available to one or two application types and whatever the standard database for applications. There are many things you can do to make new developer onboarding process easier on the new PaaS. Rather than presenting developers with a variety of choices, it helps to give what’s necessary for them and introducing new application types, data bases and other platform services and then rollout new ones as their experience grow. From a security perspective this is important as you can enable different application types when you have a formal process of auditing and reviewing in place.

Using TCPMon With Secured Services

You might have noticed when you secure a service in the WSO2 platform, that service is only exposed through https for security. This from a security standpoint is critical for any deployment. However, it’s a minor inconvenience when you’re a developer and developing a secured service and want to find out what type of messages are exchanged. When you use WS-Security these are applied as a special <wsse:Security …> header. Since you’ll be getting a generic error related to security, you need to see what’s going on to the server and what’s coming back. TCPMon is an awesome tool to do that but it can only do http.

We can do this easily by using stunnel. Using stunnel we can create a secure tunnel to the WSO2 server we’re testing and expose an HTTP endpoint. Now, you can put TCPMon between your client and stunnel. This gives us full exposure to the SOAP messages being exchanged.

Running stunnel

Installing this on any Linux is very starightforward with whatever the package management system. On a Mac, I used brew to install it.

1
$ brew install stunnel

After that it’ll be searching for a file called stunnel.conf by default. My stunnel config below,

1
2
3
4
5
6
7
8
9
client=yes
verify=0
debug=7
pid=/usr/local/var/run/stunnel.pid

[my-https]
accept = 8080
connect = localhost:9443
TIMEOUTclose = 0

Added the debug=7 property to get debug logs into syslog (/var/log/system.log). Also had to give a location of the pidfile. Important config to note here is the my-https config part. There we’ve specified a listen port (8080) and the host:port combination the connections should tunnel to.

Use TCPMon through stunnel!

Use TCPMon to create a listen port and use that in the client as the service endpoint. Here’s what you’ll see in TCPMon.

SSO Between WSO2 Servers - 8 Easy Steps

Follow these 8 easy steps to configure SAML2 Single Sign On between multiple WSO2 servers. Here I’ll be using Identity Server 4.6.0 and Application Server 5.2.1. You can add multiple servers such as ESB, DSS and so on. This assumes you’re running each server with a port offset on a single machine. You can leave Identity Server port offset untouched so it’ll be running on 9443 by default. Go to <WSO2_SERVER>/repository/conf/carbon.xml and increase the <Offset> by one for each server.

First of all we’re going to share governance registry space between multiple servers. This way, when you create a tenant, information such as the keystores that’s generated for that tenant will be accessible across multiple servers.

Step 1 - Creating databases

1
2
3
4
5
mysql> create database ssotestregistrydb;
Query OK, 1 row affected (0.00 sec)

mysql> create database ssotestuserdb;
Query OK, 1 row affected (0.00 sec)

Step 2 - Create DB schema

Create the schema for this using <WSO2_SERVER>/dbscripts/mysql.sql

1
2
$ mysql -u root -p ssotestregistrydb < wso2as-5.2.1/dbscripts/mysql.sql
$ mysql -u root -p ssotestuserdb < wso2as-5.2.1/dbscripts/mysql.sql

Note that it’s the same schema for both databases. This is because the database script have table definitions for both registry and user management. Later on you can create only registry related tables (starts with prefix REG_) and user management related tables (starts with UM_) when you do a production deployment.

Step 3 - Adding newly created DBs as data sources

Open <WSO2_IS>/repository/conf/datasources/master-datasources.xml and add configs for Registry and user mgt. Put the same 2 data sources to <WSO2_AS>/repository/conf/datasources/master-datasources.xml as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
<datasource>
    <name>WSO2_CARBON_DB_REGISTRY</name>
    <description>The datasource used for registry and user manager</description>
    <jndiConfig>
        <name>jdbc/WSO2CarbonDBRegistry</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:mysql://localhost:3306/ssotestregistrydb</url>
            <username>root</username>
            <password>root</password>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>

<datasource>
    <name>WSO2_CARBON_DB_USERMGT</name>
    <description>The datasource used for registry and user manager</description>
    <jndiConfig>
        <name>jdbc/WSO2CarbonDBUserMgt</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:mysql://localhost:3306/ssotestuserdb</url>
            <username>root</username>
            <password>root</password>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>

Make sure to copy MySQL JDBC driver to all WSO2 servers <WSO2_SERVER>/repository/components/lib

Step 4 - Change user management DB

In both Identity Server and Application Server, open up <WSO2_SERVER>/repository/conf/user-mgt.xml,

1
2
3
4
5
6
7
8
9
10
<Configuration>
<AddAdmin>true</AddAdmin>
        <AdminRole>admin</AdminRole>
        <AdminUser>
             <UserName>admin</UserName>
             <Password>admin</Password>
        </AdminUser>
    <EveryOneRoleName>everyone</EveryOneRoleName>
    <Property name="dataSource">jdbc/WSO2CarbonDBUserMgt</Property>
</Configuration>

Step 5 - LDAP configuration

Copy the LDAP config from <WSO2_IS>/repository/conf/user-mgt.xml, change the LDAP host/port and put it to <WSO2_AS>/repository/conf/user-mgt.xml. Comment out JDBC user store manager from AS user-mgt.xml. This way, we’re pointing Application Server to the embedded LDAP user store that comes with Identity Server. This will act as the user store in this setup. Why you need a separate relational data store? While LDAP will hold all the user and roles, MySQL DB will have all permissions related to users and roles.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
<UserStoreManager class="org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager">
    <Property name="TenantManager">org.wso2.carbon.user.core.tenant.CommonHybridLDAPTenantManager</Property>
    <Property name="defaultRealmName">WSO2.ORG</Property>
    <Property name="kdcEnabled">false</Property>
    <Property name="Disabled">false</Property>
    <Property name="ConnectionURL">ldap://localhost:10389</Property>
    <Property name="ConnectionName">uid=admin,ou=system</Property>
    <Property name="ConnectionPassword">admin</Property>
    <Property name="passwordHashMethod">SHA</Property>
    <Property name="UserNameListFilter">(objectClass=person)</Property>
    <Property name="UserEntryObjectClass">identityPerson</Property>
    <Property name="UserSearchBase">ou=Users,dc=wso2,dc=org</Property>
    <Property name="UserNameSearchFilter">(&amp;(objectClass=person)(uid=?))</Property>
    <Property name="UserNameAttribute">uid</Property>
    <Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>
    <Property name="ServicePasswordJavaRegEx">^[\\S]{5,30}$</Property>
    <Property name="ServiceNameJavaRegEx">^[\\S]{2,30}/[\\S]{2,30}$</Property>
    <Property name="UsernameJavaScriptRegEx">^[\S]{3,30}$</Property>
    <Property name="UsernameJavaRegEx">[a-zA-Z0-9._-|//]{3,30}$</Property>
    <Property name="RolenameJavaScriptRegEx">^[\S]{3,30}$</Property>
    <Property name="RolenameJavaRegEx">[a-zA-Z0-9._-|//]{3,30}$</Property>
    <Property name="ReadGroups">true</Property>
    <Property name="WriteGroups">true</Property>
    <Property name="EmptyRolesAllowed">true</Property>
    <Property name="GroupSearchBase">ou=Groups,dc=wso2,dc=org</Property>
    <Property name="GroupNameListFilter">(objectClass=groupOfNames)</Property>
    <Property name="GroupEntryObjectClass">groupOfNames</Property>
    <Property name="GroupNameSearchFilter">(&amp;(objectClass=groupOfNames)(cn=?))</Property>
    <Property name="GroupNameAttribute">cn</Property>
    <Property name="SharedGroupNameAttribute">cn</Property>
    <Property name="SharedGroupSearchBase">ou=SharedGroups,dc=wso2,dc=org</Property>
    <Property name="SharedGroupEntryObjectClass">groupOfNames</Property>
    <Property name="SharedGroupNameListFilter">(objectClass=groupOfNames)</Property>
    <Property name="SharedGroupNameSearchFilter">(&amp;(objectClass=groupOfNames)(cn=?))</Property>
    <Property name="SharedTenantNameListFilter">(objectClass=organizationalUnit)</Property>
    <Property name="SharedTenantNameAttribute">ou</Property>
    <Property name="SharedTenantObjectClass">organizationalUnit</Property>
    <Property name="MembershipAttribute">member</Property>
    <Property name="UserRolesCacheEnabled">true</Property>
    <Property name="UserDNPattern">uid={0},ou=Users,dc=wso2,dc=org</Property>
    <Property name="RoleDNPattern">cn={0},ou=Groups,dc=wso2,dc=org</Property>
    <Property name="SCIMEnabled">true</Property>
    <Property name="MaxRoleNameListLength">100</Property>
    <Property name="MaxUserNameListLength">100</Property>
</UserStoreManager>

Step 6 - Mount governance registry space

This configuration share the governance registry space between WSO2 servers. Having a common governance registry space is mandatory when you create tenants because tenant specific keystores are written and kept in registry.

In <WSO2_IS>/repository/conf/registry.xml,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<dbConfig name="wso2registry_gov">
    <dataSource>jdbc/WSO2CarbonDBRegistry</dataSource>
</dbConfig>

...

<remoteInstance url="https://localhost:9443/registry">
    <id>govregistry</id>
    <dbConfig>wso2registry_gov</dbConfig>
    <readOnly>false</readOnly>
    <enableCache>true</enableCache>
    <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/governance" overwrite="true">
    <instanceId>govregistry</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>

Add the same configuration to <WSO2_AS>/repository/conf/registry.xml as well. So the Application Server is also pointing to the same governance space.

Step 7 - Install SSO management feature in Identity Server

Start Identity Server, login as admin/admin and go to Configure -> Features. Add a new repository.

Repo URL - http://dist.wso2.org/p2/carbon/releases/turing/

Uncheck “Group features by category” checkbox

Seach by features with the string “stratos”

Select “Stratos - Stratos SSO Management - 2.2.0” feature and install it. Click Finish. Restart Identity Server

Step 8 - Create SSO IdP configuration

Create the file - <WSO2_IS>/repository/conf/sso-idp-config.xml following should be the content,

1
2
3
4
5
6
7
8
9
10
<SSOIdentityProviderConfig>
    <ServiceProviders>
        <ServiceProvider>
            <Issuer>carbonServer</Issuer>
            <AssertionConsumerService>https://localhost:9444/acs</AssertionConsumerService>
            <SignResponse>true</SignResponse>
            <EnableAttributeProfile>true</EnableAttributeProfile>
        </ServiceProvider>
    </ServiceProviders>
</SSOIdentityProviderConfig>

You should have a <ServiceProvider>…</ServiceProvider> config for each WSO2 server you’re using

You can test SSO by logging into Identity Server as admin/admin (this is the super user) and creating a new tenant by going to Configure -> Add New Tenant. Then try to login to Application Server. You’ll be redirected to Identity Server login page. Now login as the tenant admin user you just created. If you want to add additional servers like ESB, DSS all you have to do is get the same configuration you did for Application Server here. Replace with the correct port and the Issuer.

Things to Know Before Moving

Moving can be easy or very stressful depending on when and how you do it. These are somethings I wish I knew before. I’m jotting them down incase it might be useful for someone.

First thing you have to do when you move is to get a local telephone number if you don’t have one already. I didn’t believe the importance of having a telephone until I walked down the path of getting different things setup. Things like cable/internet, electricity are easily looked up from the phone number (and address too of course). Phones with multi-year contracts are a thing of the past. Most carriers sell sim cards with prepaid/post paid plans which are very easy to move/upgrade later if you’re unsatisfied or want to move to a plan with more minutes/data.

As soon as you get a phone the next thing is to apply for a social security number. Social security number is important for getting salary/medical insurance docs sorted. Usually the office opens at 9am. Going around 8.45am would be ideal. I had to come back 3 times because there were so many people and a lot of waiting time. Social security takes about 2 weeks to come by mail.

Opening a bank account is the next important thing. The bank I chose to do banking with required to have social security in order to open an account. There might be banks that doesn’t require having an SSN so please look around for that. If you cannot open an account that means you’ll have to wait at least 2 weeks to get it done because of the SSN. This created a bunch of headaches for me so you’re better off choosing a bank that doesn’t require having an SSN at first and may be move later.

Having done all that then finding a place to stay is the most challenging. There are tens of neighborhoods to choose from. An easy rule of thumb to follow is finding a place near the work place. There are many factors to consider and everyone has their set of preferences like whether you want city life, amenities, security, crime rate, shops and so on. There are many sites out there that let you search on different neighborhoods as well as on specific price ranges. Having local friends who’s been living nearby for sometime can give valuable advice too. Once you find a place then you have to keep a security deposit and the first month’s rent as the movein charge. This, in most places have to come in as a chashiers cheque or a money order. You can’t get a cashiers cheque without having a bank account. So if you’re in a hurry to find a place then better to have a bank account somewhere. When you get a place and sign the lease, usually 12 months (can be less depending on the property and location) if you have to move out before the lease end period, a lease break fee will be charged and it’s HUGE (in some cases). So make sure to ask that in advance if possible. Also when you move out (if it’s an apartment) they’ll be charging you for carpets/paints/doors/windows/washrooms for their condition and state. Depending on the condition it’s going to be several hundred dollars, even if it “looks almost new” and you’ve barely used them or was in that place for a short time. So discussing these in advance and being ready for those helps.

When you first come, you’ll have to arrange some transportation to go around places. Once you’re ready to get a vehicle for a longer term, there are many options. There are many different car dealers and even if you’re a seasoned car buyer, chances of getting caught up in scammers the car dealers put through is very high. Dealers get a sense of who you are just by looking at you and will try to get more money out of you. When you’re in a dealership and your see something you like then from that point on you have to be extra careful. Even though they ask your information to show around doesn’t mean you have to buy it immediatly agreeing to whatever they say. You can walk away until you put your signature down in a lease. Here, again whether you want to buy or lease depends on your requirements. Remember that when you lease, there’s a cap for miles you can drive the car on an yearly basis. Starts from about 10k miles and increment in 2.5k miles or so. When you want to drive more miles your monthly premium will go up. So if you’re in for the long haul buying will be a cheaper option. Don’t forget to do your homework by going to many sites out there which gives you valuable information about leasing/buying, which car to get, cost of ownership, gas, maintenance etc… Also importantly look for ratings/reviews of a dealership before.

Medical insurance is another critical thing that to get sorted. When you’re in an unknown place how to get medical help is very valuable info. Here ofcourse without medical insurance it can be hell. I’ve been to an emergency care facility where we were politely asked to go elsewhere middle of the night because at that time we didn’t have medical insurance. You walk into an emergency care because it’s a fucking emergency. Later that day we found a place where they treated us so well without any questions or fuss. Anyway, have the medical sorted ASAP. If you’re asked to choose a medical plan that can be very hard and tricky. When you think about it, it’s all easy you think how can something as cut and dry as this will be so difficult. One problem that I had to deal with is insurance brokers don’t tell you what you need to know. It’s almost as if they’re trying to make it deliberately complex and the plans and the literature around that is so complex and convoluted to follow. If you have just one option then all is easy.

When you have to move states then some of these you have to do all over again. States differ in salary/taxes and even little things around how get a house/apartment. When you have to do that it’s better to carry all of your docs with you because you don’t know what documents people might be asking. I had to get a social security confirmation letter which as I heard later, is going to be phased out soon. It’s better to ask the house/apartment people in advance what sort of documents you have to submit and prepare for that so it’s one less hassel.