Sample distributed deployment of WSO2 API Manager

In this post, we will be setting up a distributed deployment of WSO2 API Manager (v2.1.0) in a local machine. The official documentation of WSO2 APIM provides comprehensive details about the architecture and the components so let me just dive straight into the tricky configuration section.

Source: docs.wso2.com

Download the required components

Download the WSO2 API Manager and extract it.
Download MySQL Server and install it.
Download MySQL JDBC Driver and extract it.
Copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) into the downloaded API Manager in <PRODUCT_HOME>/repository/components/lib

Name the five profiles

Make 5 copies of the API Manager and name the directories accordingly.
Remove unnecessary webapps from each node (<PRODUCT_HOME>/repository/deployment/server/webapps) so that the following required webapps are present in each.

Publisher	`am#sample#pizzashack#v1` `api#am#publisher#v0.11` `authenticationendpoint`
Store	`api#am#store#v0.11` `authenticationendpoint`
Gateway Manager	`am#sample#pizzashack#v1` `api#am#admin#v0.11` `authenticationendpoint`
Key Manager	`Authenticationendpoint` `client-registration#v0.11` `oauth2` `throttle#data#v1`
Traffic Manager	`shindig`

Configure carbon.xml

Rename the HostName and MgtHostName of each node in each <PRODUCT_HOME>/repository/conf/carbon.xml

HINT: You can open the entire directory in IntelliJ and use shortcuts such as Ctrl+Shift+n to easily navigate to required files.

Change the port offset of each profile by changing the <Offset> entry under <Ports> as follows.

I will be using the following port offsets and local mysql databases for the deployment. Note that I allow Traffic Manager to use the default port which is slightly different from the guidelines in the official WSO2 documentation.

WSO2 Server instance	Port Offset	Port Value
Traffic Manager	0	9443
Gateway	1	9444
Publisher	2	9445
Store	3	9446
Key Manager	4	9447

Create the databases

Create a MySQL user using the following command
mysql-u username -p
Provide a password when prompted. In this post, I will use ‘admin’ as the username and ‘admin123’ as the password and use local databases.
Create the databases apimgtdb, userdb, regdb, statdb and mbstoredb by following the guidelines in the tutorial (Step 2.6).

Configure master-datasources.xml

In Key Manager, Publisher and Store profiles, add/modify the WSO2AM_DB and WSO2UM_DB entries in master-datasources.xml file found in <PRODUCT_HOME>/repository/conf/datasources/master-datasources.xml

Make sure to update the username and password of the database user and modify the database URL if a remote database is used.

<datasource>
 <name>WSO2AM_DB</name>
 <description>The datasource used for the API Manager database</description>
 <jndiConfig>
   <name>jdbc/WSO2AM_DB</name>
 </jndiConfig>
 <definition type="RDBMS">
   <configuration>
     <url>jdbc:mysql://localhost:3306/apimgtdb?autoReconnect=true</url>
     <username>admin</username>
     <password>admin123</password>
     <defaultAutoCommit>false</defaultAutoCommit>
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
     <maxActive>50</maxActive>
     <maxWait>60000</maxWait>
     <testOnBorrow>true</testOnBorrow>
     <validationQuery>SELECT 1</validationQuery>
     <validationInterval>30000</validationInterval>
   </configuration>
 </definition>
</datasource>

<datasource>
 <name>WSO2UM_DB</name>
 <description>The datasource used by user manager</description>
 <jndiConfig>
   <name>jdbc/WSO2UM_DB</name>
 </jndiConfig>
 <definition type="RDBMS">
   <configuration>
     <url>jdbc:mysql://localhost:3306/userdb?autoReconnect=true</url>
     <username>admin</username>
     <password>admin123</password>
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
     <maxActive>50</maxActive>
     <maxWait>60000</maxWait>
     <testOnBorrow>true</testOnBorrow>
     <validationQuery>SELECT 1</validationQuery>
     <validationInterval>30000</validationInterval>
   </configuration>
 </definition>
</datasource>

In Publisher and Store profiles add/modify the WSO2REG_DB and WSO2AM_STATS_DB entries in master-datasources.xml.

<datasource>
 <name>WSO2REG_DB</name>
 <description>The datasource used by the registry</description>
 <jndiConfig>
   <name>jdbc/WSO2REG_DB</name>
 </jndiConfig>
 <definition type="RDBMS">
   <configuration>
     <url>jdbc:mysql://localhost:3306/regdb?autoReconnect=true</url>
     <username>admin</username>
     <password>admin123</password>
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
     <maxActive>50</maxActive>
     <maxWait>60000</maxWait>
     <testOnBorrow>true</testOnBorrow>
     <validationQuery>SELECT 1</validationQuery>
     <validationInterval>30000</validationInterval>
   </configuration>
 </definition>
</datasource>
  
<datasource>
 <name>WSO2AM_STATS_DB</name>
 <description>The datasource used for getting statistics to API Manager</description>
 <jndiConfig>
   <name>jdbc/WSO2AM_STATS_DB</name>
 </jndiConfig>
 <definition type="RDBMS">
   <configuration>
     <url>jdbc:mysql://localhost:3306/statdb?autoReconnect=true</url>
     <username>admin</username>
     <password>admin123</password>
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
     <maxActive>50</maxActive>
     <maxWait>60000</maxWait>
     <testOnBorrow>true</testOnBorrow>
     <validationQuery>SELECT 1</validationQuery>
     <validationInterval>30000</validationInterval>
   </configuration>
 </definition>
</datasource>

In Traffic Manager profile, modify the WSO2_MB_STORE_DB entry as follows:

<datasource>
 <name>WSO2_MB_STORE_DB</name>
 <description>The datasource used for message broker database</description>
 <jndiConfig>
   <name>WSO2MBStoreDB</name>
 </jndiConfig>
 <definition type="RDBMS">
   <configuration>
     <url>jdbc:mysql://localhost:3306/mbstoredb?autoReconnect=true</url>
     <username>admin</username>
     <password>admin123</password>
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
     <maxActive>50</maxActive>
     <maxWait>60000</maxWait>
     <testOnBorrow>true</testOnBorrow>
     <validationQuery>SELECT 1</validationQuery>
     <validationInterval>30000</validationInterval>
     <defaultAutoCommit>false</defaultAutoCommit>
   </configuration>
 </definition>
</datasource>

Configure user-mgt.xml

In Key Manager, Publisher and Store profiles, modify the user-mgt.xml file found in <PRODUCT_HOME>/repository/conf/user-mgt.xml.

<configuration> 
...
    <Property name="dataSource">jdbc/WSO2UM_DB</Property>
</configuration>
  
<UserStoreManager class="org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager">
    <Property name="TenantManager">org.wso2.carbon.user.core.tenant.JDBCTenantManager</Property>
    <Property name="ReadOnly">false</Property>
    <Property name="MaxUserNameListLength">100</Property>
    <Property name="IsEmailUserName">false</Property>
    <Property name="DomainCalculation">default</Property>
    <Property name="PasswordDigest">SHA-256</Property>
    <Property name="StoreSaltedPassword">true</Property>
    <Property name="ReadGroups">true</Property>
    <Property name="WriteGroups">true</Property>
    <Property name="UserNameUniqueAcrossTenants">false</Property>
    <Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>
    <Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>
    <Property name="UsernameJavaRegEx">^[^~!#$;%^*+={}\\|\\\\&lt;&gt;,\'\"]{3,30}$</Property>
    <Property name="UsernameJavaScriptRegEx">^[\S]{3,30}$</Property>
    <Property name="RolenameJavaRegEx">^[^~!#$;%^*+={}\\|\\\\&lt;&gt;,\'\"]{3,30}$</Property>
    <Property name="RolenameJavaScriptRegEx">^[\S]{3,30}$</Property>
    <Property name="UserRolesCacheEnabled">true</Property>
    <Property name="MaxRoleNameListLength">100</Property>
    <Property name="MaxUserNameListLength">100</Property>
    <Property name="SharedGroupEnabled">false</Property>
    <Property name="SCIMEnabled">false</Property>
</UserStoreManager>

Configure registry.xml

In Publisher and Store nodes, add the following in registry.xml found in <PRODUCT_HOME>/repository/conf/registry.xml

Make sure to update the username and URL in <cacheId>

NOTE: Do not modify <dbConfig name="wso2registry"> because it is a compulsory configuration that must exist in the file. Add the new entry beneath it.

<dbConfig name="govregistry">
  <dataSource>jdbc/WSO2REG_DB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost">
   <id>gov</id>
   <cacheId>admin@jdbc:mysql://localhost:3306/regdb</cacheId>
   <dbConfig>govregistry</dbConfig>
   <readOnly>false</readOnly>
   <enableCache>true</enableCache>
   <registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/governance" overwrite="true">
   <instanceId>gov</instanceId>
   <targetPath>/_system/governance</targetPath>
</mount>
<mount path="/_system/config" overwrite="true">
   <instanceId>gov</instanceId>
   <targetPath>/_system/config</targetPath>
</mount>

Add the following entry in <indexingConfiguration>
```
<skipCache>true</skipCache>
```

Configure api-manager.xml in Key Manager

Open api-manager.xml found in <PRODUCT_HOME>/repository/conf/api-manager.xml

Modify <ServerURL> in <APIGateway> as follows:

<ServerURL>https://localhost:9444/services/</ServerURL>

Modify the <APIKeyValidator> as follows. Note that I left <ThriftServerPort> commented.

<APIKeyValidator>   
     
    <KeyValidatorClientType>ThriftClient</KeyValidatorClientType>  
    <EnableThriftServer>true</EnableThriftServer>
    <ThriftServerHost>localhost</ThriftServerHost>

    ...
</APIKeyValidator>

Make the following changes in <ThrottlingConfigurations>.

<ThrottlingConfigurations>
    <EnableAdvanceThrottling>true</EnableAdvanceThrottling>
    <DataPublisher>
        <Enabled>true</Enabled>
        <Type>Binary</Type>
        <ReceiverUrlGroup>tcp://${carbon.local.ip}:9611}</ReceiverUrlGroup>
        <AuthUrlGroup>ssl://${carbon.local.ip}:9711</AuthUrlGroup>
        ...
    </DataPublisher>
    ...
</ThrottlingConfigurations>

Configure api-manager.xml in Publisher

Open api-manager.xml found in <PRODUCT_HOME>/repository/conf/api-manager.xml

Modify the <AuthManager> as follows:

    <AuthManager> 
        <ServerURL>https://localhost:9447/services/</ServerURL> 
        <Username>${admin.username}</Username>
        <Password>${admin.username}</Password> 
        <CheckPermissionsRemotely>false</CheckPermissionsRemotely>
    </AuthManager>

Modify <APIGateway> as follows:

    <APIGateway>
        <Environments>  
            <Environment type="hybrid" api-console="true">
                <Name>Production and Sandbox</Name>
                <Description>This is a hybrid gateway that handles both production and sandbox token traffic.</Description> 
                <ServerURL>https://localhost:9444/services/</ServerURL> 
                <Username>${admin.username}</Username> 
                <Password>${admin.username}</Password> 
                <GatewayEndpoint>http://localhost:8281,https://localhost:8244</GatewayEndpoint>
            </Environment>
        </Environments>
    </APIGateway>

Modify <ThrottlingConfigurations> as follows:

    <ThrottlingConfigurations>
        <EnableAdvanceThrottling>true</EnableAdvanceThrottling>
        <DataPublisher>
            <Enabled>false</Enabled>
            ....
        </DataPublisher>
        <PolicyDeployer>
            <ServiceURL>https://localhost:9444/services/</ServiceURL>
            <Username>${admin.username}</Username>
            <Password>${admin.password}</Password>
        </PolicyDeployer>
        <BlockCondition>
            <Enabled>false</Enabled> 
            ....
        </BlockCondition>
        <JMSConnectionDetails>
            <Enabled>false</Enabled>
            .... 
        </JMSConnectionDetails>
        <JMSEventPublisherParameters>
                <java.naming.factory.initial>org.wso2.andes.jndi.PropertiesFileInitialContextFactory</java.naming.factory.initial>
                <java.naming.provider.url>repository/conf/jndi.properties</java.naming.provider.url>
                <transport.jms.DestinationType>topic</transport.jms.DestinationType>
                <transport.jms.Destination>throttleData</transport.jms.Destination>
                <transport.jms.ConcurrentPublishers>allow</transport.jms.ConcurrentPublishers>
                <transport.jms.ConnectionFactoryJNDIName>TopicConnectionFactory</transport.jms.ConnectionFactoryJNDIName>
        </JMSEventPublisherParameters>
        ....
    </ThrottlingConfigurations>

Modify <APIStore> as follows:

    <APIStore>
        <DisplayURL>true</DisplayURL>
        <URL>https://localhost:9446/store</URL>
        ....
    </APIStore>

Modify <APIKeyValidator> as follows:

    <APIKeyValidator>
        ....
        <EnableThriftServer>false</EnableThriftServer>
    </APIKeyValidator>

Configure jndi.properties in Publisher

Open jndi.properties found in <PRODUCT_HOME>/repository/conf/jndi.properties

Modify it as follows:

connectionfactory.TopicConnectionFactory = amqp://admin:admin@clientid/carbon?brokerlist='tcp://localhost:5672'
topic.throttleData = throttleData

Configure api-manager.xml in Store

Open api-manager.xml found in <PRODUCT_HOME>/repository/conf/api-manager.xml

Make the following changes:

    <RevokeAPIURL>https://localhost:8244/revoke</RevokeAPIURL>
    
    <APIKeyValidator>
        <ServerURL>https://localhost:9447/services/</ServerURL>
        <Username>${admin.username}</Username>
        <Password>${admin.password}</Password>
        ....
    </APIKeyValidator>
    
    <AuthManager> 
        <ServerURL>https://localhost:9447/services/</ServerURL> 
        <Username>${admin.username}</Username> 
        <Password>${admin.password}</Password> 
        ....
    </AuthManager>

    <APIGateway> 
        <Environments> 
            <Environment type="hybrid" api-console="true">
                ....
                <ServerURL>https://localhost:9444/services/</ServerURL> 
                <Username>${admin.username}</Username> 
                <Password>${admin.password}</Password> 
                <GatewayEndpoint>http://localhost:8281,https://localhost:8244</GatewayEndpoint>
            </Environment>
            ....
        </Environments>
    </APIGateway>

    <APIKeyValidator>
        .... 
        <EnableThriftServer>false</EnableThriftServer>
    </APIKeyValidator>

Modify <ThrottlingConfigurations> as follows:

<ThrottlingConfigurations>
        <EnableAdvanceThrottling>true</EnableAdvanceThrottling>
        <DataPublisher>
            <Enabled>false</Enabled>
        ……………………
        </DataPublisher>
        …………………
        <BlockCondition>
            <Enabled>false</Enabled>
        ………………………
        </BlockCondition>
        <JMSConnectionDetails>
            <Enabled>false</Enabled>
         …………………………………
        </JMSConnectionDetails>
     ………………………………
</ThrottlingConfigurations>

Configure Traffic Manager

Replace the <PRODUCT_HOME>/repository/conf/registry.xml file with the <PRODUCT_HOME>/repository/conf/registry_TM.xml file.
Replace the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file with the <PRODUCT_HOME>/repository/conf/axis2/axis2_TM.xml file.
Remove all the existing Jaggery apps. Do this by removing all the contents from the <PRODUCT_HOME>/repository/deployment/server/jaggeryapps directory.

In api-manager.xml, disable Thrift Server as follows:

    <APIKeyValidator>
        ....
        <EnableThriftServer>false</EnableThriftServer>
    </APIKeyValidator>

Configure api-manager.xml in Gateway

Open api-manager.xml found in <PRODUCT_HOME>/repository/conf/api-manager.xml

Modify <APIKeyValidator> as follows:

    <APIKeyValidator>
        <ServerURL>https://localhost:9447/services/</ServerURL>
        <Username>${admin.username}</Username>
        <Password>${admin.password}</Password>
        ....
        <KeyValidatorClientType>ThriftClient</KeyValidatorClientType>
        <EnableThriftServer>false</EnableThriftServer>
        <ThriftClientPort>10401</ThriftClientPort> 
        <ThriftServerHost>localhost</ThriftServerHost>
    </APIKeyValidator>

Configure throttling as follows:

<ThrottlingConfigurations>
        <EnableAdvanceThrottling>true</EnableAdvanceThrottling>
        <DataPublisher>
            <Enabled>true</Enabled>
            <Type>Binary</Type>
            <ReceiverUrlGroup>tcp://localhost:9611</ReceiverUrlGroup>
            <AuthUrlGroup>ssl://localhost:9711</AuthUrlGroup>
        ……………………
        </DataPublisher>
        <PolicyDeployer> 
            <ServiceURL>https://localhost:9443/services/</ServiceURL>
        ………………
        </PolicyDeployer>
        ………………
        <JMSConnectionDetails>
            <Enabled>true</Enabled>
            <ServiceURL>tcp://localhost:5672</ServiceURL>
        …………
        </JMSConnectionDetails>
</ThrottlingConfigurations>

Comment out the following:

<JMSEventPublisherParameters>
    <java.naming.factory.initial>org.wso2.andes.jndi.PropertiesFileInitialContextFactory</java.naming.factory.initial>
    <java.naming.provider.url>repository/conf/jndi.properties</java.naming.provider.url>
    <transport.jms.DestinationType>topic</transport.jms.DestinationType>
    <transport.jms.Destination>throttleData</transport.jms.Destination>
    <transport.jms.ConcurrentPublishers>allow</transport.jms.ConcurrentPublishers>
    <transport.jms.ConnectionFactoryJNDIName>TopicConnectionFactory</transport.jms.ConnectionFactoryJNDIName>
</JMSEventPublisherParameters>

Configure axis2.xml and WebSocketInboundEndpoint.xml in Key Manager, Traffic Manager, Publisher and Store

In profiles other than Gateway, comment out the following in <PRODUCT_HOME>/repository/conf/axis2/axis2.xml

<transportSender name="ws" class="org.wso2.carbon.websocket.transport.WebsocketTransportSender">
    <parameter name="ws.outflow.dispatch.sequence" locked="false">outflowDispatchSeq</parameter>
    <parameter name="ws.outflow.dispatch.fault.sequence" locked="false">outflowFaultSeq</parameter>
</transportSender>

In profiles other than Gateway, delete the WebSocketInboundEndpoint.xml found in <PRODUCT_HOME>/repository/deployment/server/synapse-configs/default/inbound-endpoints/WebSocketInboundEndpoint.xml

Start the cluster

To start the cluster using profiles, run the following commands in terminal of the respective bin folders. (Eg. <PRODUCT_HOME>/bin). It is recommended to start the Traffic Manager first.

sh wso2server.sh -Dprofile=traffic-manager

sh wso2server.sh -Dprofile=api-publisher

sh wso2server.sh -Dprofile=api-key-manager

sh wso2server.sh -Dprofile=api-store

sh wso2server.sh -Dprofile=gateway-manager

1 comment:

LiepinshJuly 9, 2019 at 5:14 AM
Hey!
May I ask about this:
"Configure carbon.xml
Rename the HostName and MgtHostName of each node in each /repository/conf/carbon.xml"

Question is, should MgtHostName on each server point out to https://wsoapim:9444/carbon/ ? I.e. as pe rmanual "Host name to be used for the Carbon management console" is meant the server used by /carbon access and therefore configurations and for all components point to same server?
Thanks in advance.

Com Exile

Sample distributed deployment of WSO2 API Manager

Download the required components

Name the five profiles

Configure carbon.xml

Create the databases

Configure master-datasources.xml

Configure user-mgt.xml

Configure registry.xml

Configure api-manager.xml in Key Manager

Configure api-manager.xml in Publisher

Configure jndi.properties in Publisher

Configure api-manager.xml in Store

Configure Traffic Manager

Configure api-manager.xml in Gateway

Configure axis2.xml and WebSocketInboundEndpoint.xml in Key Manager, Traffic Manager, Publisher and Store

Start the cluster

About the Author

Sachithra Dangalla

1 comment:

Label Cloud

Popular Posts

StackOVerflow profile

Blog Archive

Sachithra Dangalla

Contact me :)

About Us

Search This Blog

Tweet Tweet

Popular Posts

Newsletter

Subscribe Our Newsletter