Sizing of ecommerce Site - Hybris

The sizing depends on the type of machine hardware/softwares used including os. The following recomendation can be applied to any of the ecommerce site or in general any web site.

The following layers are there for a normal medium level ecommerce site and hence separate sizing has to be applied to each layer. This is based on hybris as the platform but it can be applied to most of the ecommerce platforms

•  Web Server Cluster 
•  App Server Cluster 
•  Search Server Cluster
•  Index Server Cluster
•  DB Server 

 Web Server Cluster
Since the web server is using few resources compared to the application server, web servers are often over sized in projects and hardly use a high percentage of available resources in production.
Compression and SSL offloading are CPU resource intensive, while concurrency has a correlation with memory
Core sizing
Cores = (Concurrent requests/second) * Task Time(Time taken to compress and do ssl for a request . 10 MS is the average time to compress a request)

The concurrent requests/seconds can be calculated
            from prior experience performance test data from old projects
            using the ab tool http://httpd.apache.org/docs/2.4/programs/ab.html
            using the formula
                          requests per second  = (No Page Request + Other Requests in a page) *  
                          Page views per seconds.
                     For eg. if a site is having a single page that contains a total of 39 other requests like js,
                     css, image etc and 25 page view per seconds then the
                     request per seconds = (1+39)*25=1000 requests per seconds

Concurrent requests/second = 800 and TaskTime is 10MS cores = 800* 10MS=800*.001S=8
Concurrent requests/second = 1600 and TaskTime is 10MS cores = 1600* 10MS=800*.001S=16
Memory sizing
RAM = Concurrent requests/second / (1/ Task time) * Apache Process size + RAM allocated to OS+ RAM allocated other processes
Among the above Task Time is the time taken to compress and secure(if ssl) a request
OS RAM - 1GB, Process Size 30MB Concurrent Request/sec 1000 and  task time of 50 ms to serve 1 request. then
RAM = (1000 /(1/50ms))*30MB + 1GB + Other Process RAM(OPR)
    = 50 *30 MB +1GB +OPR
    = 1500MB +1GB+OPR
    = 2.5GB +OPR
Sizing Disk Space
Total disk Size = Static resource size + log file size + other software size like os, Apache etc
Static resource size = media size + other static resource size like js,css,html etc
media size  = number of products * number of media items * average size of medias + other media size
Suppose there are 20000 products and 3 medias per product and average size of media is 100KB then  media size = 20000 * 3 * 100KB= 6000MB= 6GB
log file size in a second= request per seconds * average number of bytes in the access log for a request
If request per seconds is 1000, average number of bytes in the access log is 50 bytes, and need to keep the log files for 5 days and archived for 25 days as compressed
size required for log for a day  = 1000* 50 * 24 *  60 * 60 = 4.32 GB
size required for log for 5 days = 4.32 *5 =21.6GB
size required for 25 days with 85% compression = 25 * 4.32 *0.15= 16.2 GB
Total access log file size =21.6+16.2=37.8 GB
Other logs will be 50% of the access logs as an average value=37.8 *  0.5=18.9GB
So total log file size = 37.8+18.9=56.7 GB
 App Server Cluster

Core Sizing
The core sizing of app server can be calculated depends on the complexity of the pages in the site and it can be categorized as low - 10 page views/sec with a single core , medium  - 5 page views/sec with a single core
To calculate the number of page views per second, using either the concurrent users or the orders per second as an input. The formulas for those approaches are given below
 PV/sec = Current user(open user session/hr) * number of request(Avg no of request(clicks) per customer) / 3600
 PV/sec = Orders per second /(number of request(Avg no of request(clicks) per customer/Conversion rate(number of completed checkouts))
Memory Sizing
The amount of RAM available to a server has a direct impact on how big you can size your Java heap without forcing the operating system to swap memory
The server memory usage can roughly be broken down into different memory segments:

• Java heap - depends on cache size and number of request + complexity of the system
• Java non-heap memory - Java non-heap memory is composed of the perm gen in the case of Java 7 and metaspace in Java 8 as well as code cache; a good starting point is using 500MB.

• Operating System = 1 - 2 GB depends on OS as average value
• Other running processes. e.g. monitoring tools =250 MB average value

Disk Space Sizing
SAP Hybris server disk space usage can be broken down into the following categories:

• SAP Hybris Platform and SAP Hybris extensions together with custom extensions - 1.5GB to 3 GB
• Media storage of product and content related data - The same calculation that is used in the Web Server Media size calculation can be applied here as well
• Files generated as a result of cronjobs, synchronization and impex and other technical activities that require media items to be created.
• Storage of server logs such as console logs - See webserver log size creation. Logs file have a high variance and range between 1 GB to 100 GB depending on how much is logged and log rotation strategies

Reference:
https://wiki.hybris.com/display/hybrisALF/Hybris+Cluster+Sizing
https://wiki.hybris.com/display/hybrisALF/Web+Server+Sizing
http://www-01.ibm.com/support/docview.wss?uid=swg21684611

Best Selling and Top Browsed Items in Websphere Commerce

Most of the time an eCommerce site has to show the Best Selling items and the Top Browsed items in the store front. Websphere Commerce is having an OOB support for the same. One of the developers asked me how to do this and that is when I realized that it is worth mentioning that steps

• Make the following changes in the wc-server.xml file
• Enable personalization ID feature
•  Ensure that the enable attribute is set to true to the
  

• Enable persistent sessions feature
•  Ensure that the enable attribute is set to true and cookieExpiry value to a +ve value
  

• Enable the SensorEventListener(ensre enable attribute is true)
• Enable the ExperimentEvaluationEventListener(ensre enable attribute is true)
•  
• Create an Emarketing spot to show the best selling/ top browsed items
• Create a Web Activity for the Best Selling or Top Browsed Items and mention the E-Marketing spot created just above. Please note that the Best Selling & Top Browsed is limited to a Category
• Now the system will show the top browsed and best selling items in the E-marketing spot. But even after doing all these you will not be able to see any of the best selling items. To see the best selling items you need to wait for next day because by default the scheduled job runs at 2:00am. If you want to see the best selling/top browsed item immediately, you need to do the following steps 
• Browse and place orders in  multiple, different catalog entries in the target category
• Run the RaiseECEvent scheduled job(This might not be required in case top browsed items list) using Admin Console. When a customer places an order, the RaiseECEvent job asynchronously check whether the categories associated with the purchased products match against any of the defined Display Best Sellers Web activities
• Run the SaveMarketingStatistics scheduled job to save the Best Selling/Top Browsed events from memory to the DMRANKINGSTAT  table
• Run the SQL query to make sure that SendMarketingTrigger job will take all the DMACTIVITY_IDs those are there in the DMARKETINSTAT table
  

  DELETE FROM DMACTATTR WHERE DMACTIVITY_ID = 0 OR DMACTIVITY_ID = 
  (SELECT DMACTIVITY_ID FROM DMACTIVITY WHERE NAME = '')

• Run the SendMarketingTriggers job to aggreate the statics from DMRANKINGSTAT  to DMRANKINGITEM table

Reference :
http://www-01.ibm.com/support/docview.wss?uid=swg21576498
http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.admin.doc/refs/rsbtestranking.htm?lang=en

APAR information on WCS

versioninfo.sh file will provide you the feature pack fix pack information about your WCS installation. But how to get the APAR installed in a particular WCS installation? There are couple of ways you can check this

• historyInfo.sh file will provide you all the installation/uninstall information about the fix pack feature pack and APARs
• The file NIFStack.xml will contain the information about the apars installed. The location of the file is /instances//properties/version/update/backup/NIFStack.xml.
• More information about the APAR installation will be there in /logs/update/APAR_name.install 

 

IHS Access Log for WebSphere Commerce sites

While doing the production server trouble shooting for many of the issues, the investigation starts from the IHS access logs. But the default IHS log configuration will not give you enough details for troubleshooting and diagnose the root cause server. It will be more complicated when there is a cache layer exists as well. The following log level will give you all the information needed for trouble shooting. The log will provide the output in a csv format so that it will be easier for further processing if needed

LogFormat  "\"%h\",\"%{%Y-%m-%d-%H.%M.%S.000000}t\",%p,\"%H\",\"%m\",\"%U\",\"%q\",%>s,\"%b\",\"%{WAS}e\",%D,\"%{CACHED_RESPONSE}o\"" common

h - Client IP
t - Access Time
p - Port
H - Protocol
m - Request method
U - URL without query String
q - Query String
>s - Status of the request
b - size of response in bytes
{WAS}e - Name of the WAS server
D - Time taken to serve request in micro seconds
{CACHED_RESPONSE}o - The request served from dyna cashe or not

If we want to shoe the request and response cookie values then we could use the following parameter for a short period of time

{Cookie}i - Cookie request header
{Set-Cookie}o - Cookie with response

Reference : http://www.ibm.com/developerworks/library/co-websphere-access-feature/
http://httpd.apache.org/docs/2.0/mod/mod_log_config.html#formats

Facebook integration in WCS 7.0

The following step by step instructions will help some one to integrate facebook with WCS Aurora store

• Open Store Management Tool from Management Center
• Select the store in which you want to enable face book and open the store via double clicking
• Select Configuration tab
• Check the check box named "Facebook Integration" and save
• The next step is to register the site to face book. If you don't have a facebook developer a/c please register as a developer first
• Then "Create Application" button in the Facebook developer page and provide the details(You can create 'N' number of applications for each environments like testing,uat prod etc)
• It will create the AppID. Add the generated to the WCS Database via providing the sql insert into storeconf(STOREENT_ID,NAME,VALUE) values (10001,'wc.facebook.application_id', 'fb_app_id_returned');
• In the settings session on the Apps session of the Facebook developer page, select the Settings. Then press the "Add Platform" button and select type of application as website
• The system ask you to enter the url enter http://localhost/webapp/wcs/stores/servlet/en/aurora as the url and press Save changes
• Restart the server
• Now when you are going to the storefront you will see the "Connect to Facebook" link the header session
• You can see the statistics in the Dashboard page of Facebook developer

Reference : http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.aurora-starterstore.doc/concepts/csmsocfbint_aurora.htm?lang=en

Making An Attribute As NonFacetable in WCS

In Pr-Feature pack 5 of V7 to making an attribute as non facetable you need to do the following steps.

• UPDATE ATTR SET searchable=0 WHERE attr_id =?(ideally this is the only step the business user makes via management center. But this change will not make the attribute as non facetable)
• DELETE FROM ATTRDICTSRCHCONF WHERE ATTR_ID = ?(The mapping between attribute and search schema)
•  DELETE FROM SRCHATTRPROP WHERE SRCHATTR_ID in (SELECT  SRCHATTR_ID FROM SRCHATTR WHERE IDENTIFIER LIKE ?(Table defining the search index attribute like facet, rank, sort etc..)
•  DELETE FROM SRCHATTR WHERE IDENTIFIER = ? 

If you are using Feature pack 5 of V7 along with the above you need to do the following steps as well

• UPDATE ATTR SET facetable=0 WHERE attr_id=?
• DELETE FROM SRCHATTR WHERE SRCHATTR_ID IN (SELECT SRCHATTR_ID FROM FACET WHERE ATTR_ID =? )
• DELETE FROM FACET WHERE ATTR_ID = ?

Enable SQL logging in websphere commerce

Follow the steps to enable SQL logging in WCS

• Log in to WAS Administrative console
• Select Troubleshooting->logs and trace->server1->change log details levels
• Expand element com.ibm.ejs.persistences.* & enable all Messages and traces for com.ibm.ejs.persistence.EJSDBCFinder and com.ibm.ejs.persistence.EJSJDBCPersister

Precompiling JSP Files in WCS

Expand the enterprise application archive using the following command. WAS_installdir/bin/EARExpander -ear /mytmp/WC_instance.ear -operationDir /mytmp/WC_instance.expanded.ear -operation expand -expansionFlags war

· Change the JSP Engine parameters. You want to ensure that the JSP engine will not compile any JSP files for the Stores Web module at run time. This ensures that your precompiled JSP files from the WEB-INF directory is used.

o Open /mytmp/WC_instance.expanded.ear/Stores.war/ibm-web-ext.xmi in a text editor.

o Add the following line before the line

</webappext:WebAppExtension>

< xmi:id="JSPAttribute_2" name="disableJspRuntimeCompilation"

value="true"/>>

Ensure that the xmi:id of the new line is unique within the file.

o Run the JSP batch compiler using the command WAS_installDir/bin/JspBatchCompiler -ear.path /mytmp/WC_jbweb1.expanded.ear -webmodule.name Stores.war -compileToWebInf true

o Since you disabled runtime compilation in Step 3, you must review the log to ensure that all JSP files compiled without error. If some JSP files are not compiled, correct the error and run the compilation step again

Blank page encountered in IE when accessing WebSphere Commerce Accelerator

While accessing the Websphere commerce accelarator in IE as a https request, I got a blank page. I was wondering why it was giving a blank page. Firefox and Google chrome was showing the page properly though. After some research I have found out the firefox and goolge chrome ask to add the https sites to the trusted site list while accessing the site itself and IE is not doing so. So to correctly get the commerce accelarator page in IE we need to add the site in the list of trusted site using the menu option Tools -> Internet Options -> Security -> Trusted Sites

Retrieving Merchant Key From Websphere Commerce

While installing the websphere commerce server you normally provide the merchant key and that key will be used for enrypting and decrypting different values in the system. When we want to directly encrypt the value using the wcs utilities we have to provide the merchant key and if we forget the merchant key we can directly get it from the server using the following methods

• Call the method com.ibm.commerce.server.ConfigProperties.singleton().getMerchantKey()
• or Call the method com.ibm.commerce.server.WcsApp.configProperties.getMerchantKey()

The above methods are there in WCS6. WCS7 on wards you have to use the following

com.ibm.commerce.util.nc_crypt.decrypt(ConfigProperties.singleton().getValue("Instance/MerchantKey")

Change User Password in WCS

In some situations you might forget the password of a particular user in wcs and you would like to set a new password to that user. Actually the password is stored in the database in the encrypted format and the encryption keys used are the merchant key and the SALT column value in the USERREG table for the corresponding user. We can change the password using the following way

Using wcs_password utility

Go to bin folder of the WCS server and supply the command

wcs_password

The command generate the password in the encrypted format and printout in the console.

Copy the encrypted password from the console and directly update the USERREG table's LOGONPASSWORD column with the generated value and update the SALT column value with the SALT value you have given in the above command

Direct Database Update

This method can be done only if you know atleast one user's password. We have to copy the LOGONPASSWORD column and SALT column of the known user to the the unknown user

Changing Merchant Key Value in WCS

There will be lot of occasions come during your life with WCS where you want to change the merchant key. The possbile reasons are

• Because of security reasons the organization decide to change the merchant key on a regular interval
• While doing enhancement to the production site. You will be in a position to migrate the data from existing production server to the new production server. For smoother data migration the requirement is that both the servers should be having same merchant key

The steps for doing the merchant key change are given below

• Run the following command where demo is the instance name and "1234567890abcdef" is the current merchant key

  MigrateEncryptedInfo db2 demo 4 5000 1234567890abcdef abcdef1234567890

• Run the command to update the server with the data "config_ant.bat -DinstanceName= demo UpdateEAR"
• Restart the server

Now check the merchant key via mentioning the steps mentioned in the article Retrieving Merchant Key From Websphere Commerce. You will see the new merchant key

RAD Performance Tips

Preference

• When Build automatically (auto build) is turned on, it means that every time you save a file, an incremental build happens. This includes not only building (or compiling) the file, but also building the files that depends on it. It also runs validation. Turning auto build off will improve the performance of your save operations at the expense of slower builds and less up-to-date error information
• Turning Refresh workspace automatically on will start a background process that searches for changes that have been made by other processes to resources in the workspace. If changes are found, this will cause an incremental build to run if auto build is turned on, but otherwise it does not initiate any builds
• By turning off some of the capabilities, you will remove certain user interface items. This in turn will prevent you from invoking certain functions. Since Rational Application Developer uses lazy loading, any time you invoke a new function, it takes more memory to hold the classes that implement the new function
• Any editor that is active (that is, has focus) when Rational Application Developer is shut down, will become active again when Rational Application Developer is restarted. To improve the startup time of Rational Application Developer, you can select the Close all editors on exit
• Quick diff is a function that shows a line in the margin to show you what has changed. Disabling quick diff will slightly improve your performance while editing files
• Disabling all or some of the label decorations will slightly improve your performance
• The Links Builder keeps track of hyperlinks. If you have a lot of HTML files in your workspace, there can be a lot of links to keep track of. In some cases, disabling the Links Builder can greatly improve your build times
• Running all of the validators all of the time is a reasonably expensive operation. If you are finding that your builds are taking too long, you could turn off all or some of the validators
• Monitor, how much memory is being used by turning on the Show heap status preference in Rational Application Developer in case of performance degradation

Other

• Keep the projects that you intend on changing in source form, and the projects that you are just using in binary form
• As a general rule you should not have projects in your workspace that are not needed. They just cause extra overhead when building, validating and indexing your workspace
• Rational Application Developer is a very large product; a typical installation has over 100, 000 files. The defragmentation of the hard disk after installing Rational Application Developer can improve startup time
• Install only what you will use in your application development and uninstall features that you no longer require
• Periodically starting with a clean workspace, and re-importing your source will improve performance. This is quite easy to do if your projects are all stored in a source control system
• Over time, there is metadata that accumulates in the workspace (in the .metadata directory). This includes things like log files, for instance. Cleaning this up every so often means that there are less things needed to keep track of for Rational Application Developer
• Running test server on another remote machine will free up resources
• As you use more functions, more classes are loaded into memory. Rational Application Developer uses lazy loading, so the code for a particular function is not loaded until it is required. For example, if you do not usually use the XML editor, but in this session you edited one XML file, the XML editor is loaded only when it is needed. However, once it is loaded it will not be un-loaded. The only way to free this memory is to close and restart Rational Application Developer. You should close Rational Application Developer at least once a day
• Never minimize Rational Application Developer since that will initiate memory swap
• Change the -Xmaxf –Xminf parameters
• Set the heap size as

size of physical memory – memory used for OS – 100MB(Other uses)

This will be almost equivalent to 2/3 of your available memory.

• When you are working with WebSphere Test Environment (WTE) inside RAD, there will be a heap size associated with WTE as well. So put ¼ the of the total makeable max heap size for WET and the rest for RAD. To specify the WTE heap within Application Developer, go to the Server perspective Server Configuration view, and open your server instance Editor Window. Select the Environment tab and enter –Xmx..M in the Java VM Arguments window. Then save and close that server editor window. A similar change is required if you run a stand-alone WAS outside of Application Developer (instead of using the embedded WTE)
• Rational Application Developer uses shared classes to improve startup time. This is enabled by specifying the -Xshareclasses directive in the eclipse.ini file. If you need to save memory, you can remove this directive as well as the -Xscmx96m directive; however, removing these directives will increase your startup times
• It is considerably faster to restart a project instead of restarting the server

Test Server Configuration

• The Enable hot method replaces in debug mode option should always be enabled. Not only does this option allow hot method replace, but more significantly from a performance point of view, it enables "Just In Time" compilation while in debug mode. This greatly speeds up how quickly your application runs and how fast the server starts. This option only applies to debug mode, and has no effect when running the server in normal mode
• If you don't need the universal test client (it is enabled by default), you should clear its checkbox so that it wont be installed
• The Optimize server for testing and developing option should always be checked. This sets some JVM parameters that improves the startup time of the server
• Set the publishing option to never publish automatically

Test Server Admin Console Configuration

• Once the Admin Console is running, select the server (it is usually called server1; Servers->Application Servers->Server1), and ensure that the Run in development mode and Parallel start options are selected. This will provide even more server startup improvements
• Check if there are any unneeded applications that could be removed. There are a couple of small sample applications (DefaultApplication, ivtApp and query) that are installed by default. These can be safely removed
• If you know that your module does not contain any Java EE annotations, set the Metadata Complete property to true in the web.xml deployment descriptor. This will signal to WebSphere Application Server that it does not need to scan the classes and JAR files for annotations