Hybris Vertical Cluster Configuration with Apache mod_jk configuration

Hybris is having its own cluster setup and hence instead of using the application server cluster configuration the hybris cluster configuration has to be used. For the demonstration purpose I have used two instances of hybris instances in the same machine and an Apache HTTP Server as a load balancer with the help of mod_jk

Hybris Cluster Setup

• Make sure that both the instances of the hybris is having same code and using the same db. 
• Configure both instances to point to the same stand alone solr server. For reference see the link Solr Clustering and Replication in Hybris
• Since both instances are running on the same machine, to avoid the port conflict change the ports in the second instance via following link Changing default ports in hybris
• In case of vertical clustering and unicast we need to specify separate ports and if we are using other methods then same configuration has to be used in all the nodes even in case of vertical clustering
• Enter the following values in the local.properties file in both the hybris isntances 

        #clustering
         clustermode=true
         cluster.nodes.autodiscovery=true
         cluster.nodes.ping.interval=10000
         cluster.nodes.stale.timeout=30000

• Unicast

                 #unicast
                 cluster.broadcast.methods=unicast
                 cluster.broadcast.method.unicast.serveraddress=localhost
                 cluster.broadcast.method.unicast.clusternodes=localhost:9997 ; localhost:19997

• JGroups

           cluster.broadcast.methods=jgroups 

           cluster.broadcast.method.jgroups=de.hybris.platform.cluster.jgroups.JGroupsBroadcastMethod

           cluster.broadcast.method.jgroups.tcp.bind_addr=localhost

           cluster.broadcast.method.jgroups.tcp.bind_port=7800

           cluster.broadcast.method.jgroups.channel.name=hybris-broadcast

           cluster.broadcast.method.jgroups.configuration=jgroups-udp.xml

• Give the following changes in first instance
  • Add the following values in the local.properties file in case unicast for other methods no change required
    • cluster.broadcast.method.unicast.port=9997
  • Add the following values in the server.xml file of conf directory see the link HTTP Server mod_jk configuration to load balance Tomcat instances in Hybris

<Connector protocol="AJP/1.3" port="${tomcat.ajp.port}" proxyPort="${proxy.http.port}" redirectPort="${proxy.ssl.port}" useIPVHosts="${tomcat.ajp.useipv}" connectionTimeout="10000" keepAliveTimeout="10000" />

<Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">

• Give the following changes in seond instance
  • Add the following values in the local.properties file
    • cluster.broadcast.method.unicast.port=19997
  • Add the following values in the server.xml file of conf directory see the link HTTP Server mod_jk configuration to load balance Tomcat instances in Hybris

<Connector protocol="AJP/1.3" port="${tomcat.ajp.port}" proxyPort="${proxy.http.port}" redirectPort="${proxy.ssl.port}" useIPVHosts="${tomcat.ajp.useipv}" connectionTimeout="10000" keepAliveTimeout="10000" />

<Engine name="Catalina" defaultHost="localhost" jvmRoute="node2">

Apache HTTP Server Load Blancer Setup
Follow the linnk HTTP Server mod_jk configuration to load balance Tomcat instances in Hybris

Changing default ports in hybris

By default hybris runs on 9001 as http and 9002 as https port etc. If any one needs to change these ports the following instruction will helpful
The port numbers are there in the platform/project.properties files as given below
tomcat.http.port=9001
tomcat.ssl.port=9002
tomcat.ajp.port=8009
tomcat.jmx.port=9003
tomcat.jmx.server.port=9004
Copy the above properties to local.properties and change it instead of changing it on the origin

Changing Environment Variables

The default environment variables can be changed with the help of the following methods

• For a particular ant call, for example: ant all -DHYBRIS_CONFIG_DIR=
• Using set/export from the console eg. set HYBRIS_CONFIG_DIR= or export HYBRIS_CONFIG_DIR=
• platform/env.properties file is having the following environment variable
• HYBRIS_BIN_DIR=${platformhome}/../../bin
• HYBRIS_CONFIG_DIR=${platformhome}/../../config
• HYBRIS_DATA_DIR=${platformhome}/../../data
• HYBRIS_LOG_DIR=${platformhome}/../../log
• HYBRIS_TEMP_DIR=${platformhome}/../../temp/hybris
• platformhome.properties file in all the extension directory is having platformhome value defined
• conf/wrapper.conf, wrapper-debug.conf, wrapper-jprofiler.conf, wrapper-minimal.conf, wrapper-yourkitprofiler.conf files are having the following properties
• wrapper.java.additional.29=-DHYBRIS_BIN_DIR="C:\hybris6\hybris\bin"
• wrapper.java.additional.30=-DHYBRIS_CONFIG_DIR="C:\hybris6\hybris\config"
• wrapper.java.additional.31=-DHYBRIS_DATA_DIR="C:\hybris6\hybris\data"
• wrapper.java.additional.32=-DHYBRIS_LOG_DIR="C:\hybris6\hybris\log"
• wrapper.java.additional.33=-DHYBRIS_TEMP_DIR="C:\hybris6\hybris\temp\hybris" 

Reference

https://help.hybris.com/6.3.0/hcd/8be84a1686691014b4b2e69a3a01897b.html

Sizing of ecommerce Site - Hybris

The sizing depends on the type of machine hardware/softwares used including os. The following recomendation can be applied to any of the ecommerce site or in general any web site.

The following layers are there for a normal medium level ecommerce site and hence separate sizing has to be applied to each layer. This is based on hybris as the platform but it can be applied to most of the ecommerce platforms

•  Web Server Cluster 
•  App Server Cluster 
•  Search Server Cluster
•  Index Server Cluster
•  DB Server 

 Web Server Cluster
Since the web server is using few resources compared to the application server, web servers are often over sized in projects and hardly use a high percentage of available resources in production.
Compression and SSL offloading are CPU resource intensive, while concurrency has a correlation with memory
Core sizing
Cores = (Concurrent requests/second) * Task Time(Time taken to compress and do ssl for a request . 10 MS is the average time to compress a request)

The concurrent requests/seconds can be calculated
            from prior experience performance test data from old projects
            using the ab tool http://httpd.apache.org/docs/2.4/programs/ab.html
            using the formula
                          requests per second  = (No Page Request + Other Requests in a page) *  
                          Page views per seconds.
                     For eg. if a site is having a single page that contains a total of 39 other requests like js,
                     css, image etc and 25 page view per seconds then the
                     request per seconds = (1+39)*25=1000 requests per seconds

Concurrent requests/second = 800 and TaskTime is 10MS cores = 800* 10MS=800*.001S=8
Concurrent requests/second = 1600 and TaskTime is 10MS cores = 1600* 10MS=800*.001S=16
Memory sizing
RAM = Concurrent requests/second / (1/ Task time) * Apache Process size + RAM allocated to OS+ RAM allocated other processes
Among the above Task Time is the time taken to compress and secure(if ssl) a request
OS RAM - 1GB, Process Size 30MB Concurrent Request/sec 1000 and  task time of 50 ms to serve 1 request. then
RAM = (1000 /(1/50ms))*30MB + 1GB + Other Process RAM(OPR)
    = 50 *30 MB +1GB +OPR
    = 1500MB +1GB+OPR
    = 2.5GB +OPR
Sizing Disk Space
Total disk Size = Static resource size + log file size + other software size like os, Apache etc
Static resource size = media size + other static resource size like js,css,html etc
media size  = number of products * number of media items * average size of medias + other media size
Suppose there are 20000 products and 3 medias per product and average size of media is 100KB then  media size = 20000 * 3 * 100KB= 6000MB= 6GB
log file size in a second= request per seconds * average number of bytes in the access log for a request
If request per seconds is 1000, average number of bytes in the access log is 50 bytes, and need to keep the log files for 5 days and archived for 25 days as compressed
size required for log for a day  = 1000* 50 * 24 *  60 * 60 = 4.32 GB
size required for log for 5 days = 4.32 *5 =21.6GB
size required for 25 days with 85% compression = 25 * 4.32 *0.15= 16.2 GB
Total access log file size =21.6+16.2=37.8 GB
Other logs will be 50% of the access logs as an average value=37.8 *  0.5=18.9GB
So total log file size = 37.8+18.9=56.7 GB
 App Server Cluster

Core Sizing
The core sizing of app server can be calculated depends on the complexity of the pages in the site and it can be categorized as low - 10 page views/sec with a single core , medium  - 5 page views/sec with a single core
To calculate the number of page views per second, using either the concurrent users or the orders per second as an input. The formulas for those approaches are given below
 PV/sec = Current user(open user session/hr) * number of request(Avg no of request(clicks) per customer) / 3600
 PV/sec = Orders per second /(number of request(Avg no of request(clicks) per customer/Conversion rate(number of completed checkouts))
Memory Sizing
The amount of RAM available to a server has a direct impact on how big you can size your Java heap without forcing the operating system to swap memory
The server memory usage can roughly be broken down into different memory segments:

• Java heap - depends on cache size and number of request + complexity of the system
• Java non-heap memory - Java non-heap memory is composed of the perm gen in the case of Java 7 and metaspace in Java 8 as well as code cache; a good starting point is using 500MB.

• Operating System = 1 - 2 GB depends on OS as average value
• Other running processes. e.g. monitoring tools =250 MB average value

Disk Space Sizing
SAP Hybris server disk space usage can be broken down into the following categories:

• SAP Hybris Platform and SAP Hybris extensions together with custom extensions - 1.5GB to 3 GB
• Media storage of product and content related data - The same calculation that is used in the Web Server Media size calculation can be applied here as well
• Files generated as a result of cronjobs, synchronization and impex and other technical activities that require media items to be created.
• Storage of server logs such as console logs - See webserver log size creation. Logs file have a high variance and range between 1 GB to 100 GB depending on how much is logged and log rotation strategies

Reference:
https://wiki.hybris.com/display/hybrisALF/Hybris+Cluster+Sizing
https://wiki.hybris.com/display/hybrisALF/Web+Server+Sizing
http://www-01.ibm.com/support/docview.wss?uid=swg21684611

HTTP2 Support in Apache HTTP Server

The http2 support is there from Apache Http Server 2.4.17 or higher with the help of mod_http2 module
Once http server is enabled with http2 it can serve hybris that is running on an http 1.1 protocol so that the end user always get the http2 protocol. Apche can be front end with any web application via following the blog HTTP Server mod_jk configuration to load balance Tomcat instances
Steps to enable http2 protocol are given below

• Load the http2 module via adding the following lines in httpd.conf file

         LoadModule http2_module modules/mod_http2.so

• Add the protocol h2 in VirtualHost for ssl  virtual host area(assumption:already ssl is enabled)

         <VirtualHost *:443>
             ......
             Protocols h2 http/1.1
             .........
        </VirtualHost
Restart the http server and access the ssl page so that you can see the server is enabled with http2

HTTP2 Support in Tomcat

From Tomcat version 9 on wards the server support http2

Http2 protocol works only on the ssl pages and hence need to enable ssl in tomcat first via doing the following steps
Create the ssl certificate openssl req -x509 -nodes -days 1095 -newkey rsa:2048 -out ../conf/localhost.crt -keyout ../conf/localhost.key -config ../conf/openssl.cnf
Add the following session in the server.xml file
    <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="150" SSLEnabled="true" >
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
        <SSLHostConfig>
            <Certificate certificateKeyFile="conf/certificate/localhost.key"
                         certificateFile="conf/certificate/localhost.crt"
                          type="RSA" />
        </SSLHostConfig>
    </Connector>
Restart the server and go to https://localhost:8443/ the protocol used is http2
To check whether the server is returning back using http2 the following methods can be applied

• Go to the tomcat's access log file and you can see "GET / HTTP/2.0" 200
• Open chrom's developer tool and right click on the header and select protocol as well in the network tab. The protocol column shows h2 instead of http/1.1 

• Install the chrome http2/SPDY plugin to check whether the site supports http2

HTTP Server mod_jk configuration to load balance Tomcat instances in Hybris

The following configuration can be applicable to any web application that is running in tomcat proxied with Apache http server

Assumption is that already ssl is enabled in Apache refer the blog  HTTP Server Proxy Configuration for Hybris
Download the mod_jk module for the corresponding OS from http://archive.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/
Put the mod_jk.so file in the modules directory of
Create a file named workers.properties in the conf directory of apache with the following contents

worker.list=loadbalancer

worker.loadbalancer.port=8009

worker.loadbalancer.host=electronics.local

worker.loadbalancer.type=ajp13

If you are having two servers that you need to load balance then the contents of workers.properties is given below

worker.list=loadbalancer,status

# Define Node1

# modify the host as your host IP or DNS name.

worker.node1.port=8009

worker.node1.host=node1.electronics.local

worker.node1.type=ajp13

worker.node1.ping_mode=A

worker.node1.lbfactor=1

# Define Node2

# modify the host as your host IP or DNS name.

worker.node2.port=8009

worker.node2.host=node2.electronics.local

worker.node2.type=ajp13

worker.node2.ping_mode=A

worker.node2.lbfactor=1

# Load-balancing behavior

worker.loadbalancer.type=lb

worker.loadbalancer.balance_workers=node1,node2

worker.loadbalancer.sticky_session=1

# Status worker for managing load balancer

worker.status.type=status

Add the following to the httpd.conf file

LoadModule jk_module modules/mod_jk.so

<IfModule jk_module>

JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogStampFormat "[%b %d %Y - %H:%M:%S] "
JkRequestLogFormat "%w %V %T"
JkLogLevel info
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

<Location /*/WEB-INF/*>

deny from all

</Location>

JkMount / loadbalancer
JkMount /* loadbalancer

</IfModule>

 Add the following to the node that is there in httpd-ssl.conf file

JkMount /* loadbalancer

Open the tomcat's server.xml and add/change the AJP connector port with the following

<Connector protocol="AJP/1.3" port="8009" proxyPort="80" redirectPort="443" useIPVHosts="${tomcat.ajp.useipv}" connectionTimeout="10000" keepAliveTimeout="10000" /> 

<Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">

jvmRoute="node1" will add node1 to the jsessionid so that the request can be correctly identified from which node it is coming from

Restart http server
Restart tomcat
Open the following urls(assumption is that electronics.local is already mapped to the host file)

https://electronics.local/theworksstorefront/electronics/en/

http://electronics.local/hmc