Executive summary
It may come as a great surprise, but most electronic commerce enterprises are not prepared for success. Electronic commerce (e-commerce) enterprises climb a steep information technology ramp that must provide bullet-proof continuous availability, scalability for millions of users, and sophisticated user-relationship-management clickstream data warehouses in the first months of business. Ironically, a hugely successful Web site can mean either an exploding business success or an exploding business plan, depending on how well the e-commerce enterprise plans and executes its information technology infrastructure.

This paper explores the common pitfalls in the design and implementation of a successful e-commerce information architecture. After identifying the most common problems, the paper shows how to 1) architecturally guarantee continuous Web site availability and scalability, 2) successfully implement a clickstream data warehouse, and 3) create a contractual environment with technology suppliers that ensures the business success of the e-commerce enterprise. If answers to these e-commerce information technology issues are of interest to your enterprise, please read on...
 
 

Millions of users on day one
Business-to-consumer (B2C) electronic commerce enterprises are information-driven entities that have some of the most extreme information technology solution requirements found in the commercial business world. From the moment B2C e-commerce Web sites go online, they must provide the functionality, capacity, and continuous availability required for the potentially millions of users accessing Web site content. These extreme functionality, capacity and availability requirements apply whether the enterprise is a new e-commerce startup or the e-commerce presence of an existing brick-and-mortar business.

But are the millions of e-commerce site users necessarily also site customers? Obviously not梥ite users can be anyone from casual visitors, to advertising click-throughs, to targeted prospects, to actual customers. While traditional brick-and-mortar commerce enterprises typically have no easy way to record and analyze user behavior until they become customers, if even that, e-commerce enterprises can record and analyze all activities of all types of users, all of the time. See Figures 1 and 2.
 
 

Figure 1: Big differences in the types of electronically recordable user behavior

The ability to record and analyze all site user behavior in minute detail gives e-commerce enterprises a significant edge over brick-and-mortar competitors who have little direct visibility to the behavior of anyone besides actual customers.
 
 

Figure 2: e-commerce systems cover a wider range of user behavior

The increased information processing requirements of the information-rich Web e-commerce environment place special requirements on the information architecture of a B2C e-commerce enterprise. These requirements and their corresponding solution are the subject of the rest of this paper.
 
 

e-commerce information system architecture
Electronic commerce enterprises typically have five categories of business activity that are realized in up to a dozen potential business activity fulfillment mechanisms. The correspondence between the business activities and their fulfillment mechanisms is shown in the Figure 3. A particular e-commerce enterprise may not have a full suite of the fulfillment mechanisms, depending on its business model and its maturity as a business. For example, not all electronic commerce enterprises have telephone call centers. At a minimum, they require only a Web-based customer service interface. But a fully-functional e-commerce enterprise generally will have aspects of all the business activities and the fulfillment mechanisms.

Figure 3: Five e-commerce business activities梞ap to梩welve potential e-commerce business activity fulfillment mechanisms

 

The twelve e-commerce business activity fulfillment mechanisms map into six information architecture system components shown in Figure 4.
 
 

Figure 4: e-commerce business activity fulfillment mechanisms梞ap to梕-commerce information architecture system components

These e-commerce information architecture system components fall into four categories:
 

e-commerce information system architecture stakeholders
The e-commerce information architecture components have associated cadres of interested stakeholders, which are described below. See Figure 5. The first set of stakeholders is the site's user community, whose attention is garnered either by external acquisition systems, like search engines, Web ads, e-mail, or other media advertising, or by the internal content of the site's front-end Web servers and call centers. If the eyeball acquisition mechanisms or the site and call center fail to intrigue or at least be useful to users, they fail these very important stakeholders. Again we note that not all sites have call center systems, but all e-commerce sites have Web servers by definition.

The next set of stakeholders is the e-commerce enterprise's operational personnel, who are responsible for the care and feeding of the front-end Web servers, call centers, and the operational tie-ins to external user acquisition media. In many cases, e-commerce operational personnel are solely focused on these front-end systems despite the obvious need for back-end system functionality. For more enlightened e-commerce enterprises, operational personnel also have a stake in order entry/order fulfillment systems, which centrally fulfill site-oriented activities, which may include things like user registration, requests for information, downloads, product or service (stock/auction) orders, etc. Web server site content may also be distributed centrally from such a system.

e-commerce executive management and stockholders have a stake in the success of all the systems of an enterprise, but they are acutely interested in a well implemented enterprise financial management system. These systems enable up-to-the-moment financial management of the enterprise, directly linked to the actual revenue and cost streams associated with front-end and back-end operational systems. Without such a system, management is essentially flying blind from a financial perspective, and this is very dangerous, and unfortunately a very common situation with young e-commerce enterprises. A properly-implemented enterprise financial management system can also satisfy the financial reporting requirements set forth by the SEC and investment bankers, which are among the required prerequisites for successful initial public offerings.

Figure 5: e-commerce information architecture component stakeholders: user-oriented value-chain view

While enterprise financial management is a fundamental need, the clickstream/callstream data warehouse gives user relationship management stakeholders the tools they need to make the e-commerce enterprise grow and prosper over the long term. The existence of an effective user relationship management clickstream/callstream data warehouse is one of the most important long-term success differentiators in the e-commerce business sphere. These systems record all the activity of prospects and customers, whether the contact mechanism is via the Web or a call center. Examples of captured Web clickstream information include items like the referring site, which tests the effectiveness of external media user acquisition campaigns, the Web pages visited by the site user, the time spent there, and what the user might have bought. The call center user contact records what the user did in this contact environment, either by automated voice-activated telephone scripts or by contact with human call center personnel.

This automatically-collected treasure trove of user behavior information theoretically gives the e-commerce enterprise a huge advantage over brick-and-mortar competitors, who cannot easily record customer and prospect behavior. While Web server clickstream and call center log files provide the necessary source information, the resultant historical data warehouses can become extremely large, making them probably the most difficult e-commerce system to implement. But once implemented these systems affect all aspects of the enterprise and all stakeholders, including:
 

A summary of the interests of the various e-commerce stakeholders is shown in
Figure 6.

Figure 6: Key interests of e-commerce stakeholders

The principal problem with e-commerce information systems
So, what is the principal problem with this complex e-commerce information architecture landscape? The lack of focus on back-end e-commerce information systems.

Figure 7: Properly implemented back-end e-commerce systems ensure long-term success

Fundamental Business Problems

Many e-commerce startups are expert-heavy in front-end Web servers and business domain knowledge. This is good, but they often lack corresponding expertise in the details of the back-end information systems that enable critical functions like order entry and fulfillment, financial management, and the analysis of user behavior on their Web sites. See Figure 7. This lack of depth in a total e-commerce IT solution leads to the following fundamental business problems:
 
 
 

It is easy to lay blame and identify problems, but how does one construct a viable solution model that conquers these three fundamental e-commerce enterprise dilemmas? The next three sections explore solutions to each of these important problems.
 
 

Problem one: a highly available and highly scalable e-commerce operational infrastructure
Because e-commerce enterprises are information entities at their core, the problem of high availability becomes particularly acute. A highly available implementation of the information architecture of an e-commerce enterprise also needs to deliver high levels of performance, even in the face of failure, which places special performance design and implementation requirements on this highly available architecture.

Typical e-commerce enterprises take care of the first level of these requirements by sitting their Web servers at Web site-hosting service providers, like Exodus, AboveNet, Frontier GlobalCenter, and others. These site-hosting service providers typically have multiple geographically separated, secure buildings that are sited on top of central-location Internet backbone connections for fast access. The e-commerce enterprise's Web servers are placed at one or more of the Web-host service provider sites, creating fast, replicated Web server access for site users. In addition, the site-hosting service providers typically create a high availability environment for the hosted site servers, including such features as redundant Internet backbone connections, redundant uninterruptable power grids, non-water-based fire suppression, and caged-system physical security measures. An architecture diagram of this type of environment is shown in Figure 8.

Figure 8: Typical "highly available" e-commerce user site access solution, using Web site-hosting services

Multiple replicated-content Web servers reside at these sites, providing a continuous Web presence for the e-commerce enterprise. Users enjoy fast and reliable access, and operational personnel can theoretically sleep at night.

But not all of the critical e-commerce systems can be replicated as easily as the Web server front-ends. Back-end systems tend to be centralized by their very nature and this creates a new set of single-point-of-failure problems that go beyond the site and environmental redundancy provided by the site-hosting service provider.

For example, back-end order entry, processing and fulfillment systems are usually centralized. A customer who places an order from a particular Web server may later inquire about its status from another server, and all the order information needs to be stored in a centralized database so that this functionality can be delivered from any point of entry. But the system that houses this centralized order database becomes a major single point of failure that requires a redundant architecture on top of that provided by the Web hosting service provider.

Any failure in the hardware or software of the centralized back-end order system can stop an e-commerce site, and the types of 攐rders? can be anything from a purchase of physical goods, served-up advertisements, auction site bids, stock market buy and sell orders to site content information. These centralized information systems must have a redundant, clustered, highly available implementation within a particular site-hosting environment, or they become a single point of failure.

In addition to a fully redundant site-hosting architecture, it is also necessary to replicate the host site systems between at least two geographically remote locations for protection against catastrophic host site disasters, such as widespread power failures, earthquakes, storms, terrorism, and war. The replicated, multi-site strategy also enhances localized Internet site access performance, by shortening the pathway from the client to the nearest geographic server. In addition, the business model of many e-commerce enterprises causes the clickstream/callstream data warehouse to become so critical to operations that it, too, must be clustered and remotely replicated to ensure the viability of the enterprise.

Taking all of the above issues into consideration, a fully redundant, geographically replicated, high-availability e-commerce system architecture, including all front-end and back-end systems, is shown in Figure 9.

Figure 9: Fully redundant, geographically replicated Web site hosting architecture

Problem two: the design and implementation of an effective clickstream/callstream data warehouse
It is a cliché, but the Web changes everything about the design of a business-to-consumer commerce-oriented data warehouse. The wide scope of this change is best appreciated by reviewing the typical data warehouse schema of traditional brick-and-mortar retailers, a simple example of which is shown in Figure 10.

Figure 10: Traditional brick-and-mortar retail data warehouse

While traditional brick-and-mortar retail data warehouses differ greatly in the details of their specific implementations, all have some version of four key dimensions: Time, Product, Geography, and Promotion, as well as a Sales Fact table containing sales transaction data. Note that there is no notion of a customer dimension in this old-style data warehouse. Until recently, it was so difficult to capture the identity of a specific customer and his associated market basket that this key analytical dimension, perhaps the most important of all the dimensions, was left out. Nevertheless, the information contained in this type of schema has changed the face of retailing, greatly improving inventory management, store layout, and mass-media advertising effectiveness.

By using newer information technology customer identification techniques like loyalty cards or linkage to checking and credit card payment databases, forward-looking brick- and-mortar retailers have been able to add the Customer Dimension and associated market basket analytical capabilities to their data warehouses. This enhancement has driven the ongoing movement toward one-to-one Customer Relationship Management (CRM).

Customer/market-basket analysis is a great advance, but there are two classes of potential customer activity that are not captured by loyalty cards or other brick-and-mortar information system tie-ins. Sales prospects are potential customers that do comparison shopping at different stores, catalogs, and assess various advertisements. They move in and out of a particular retailing environment without leaving a trace of their activity, unless, of course, they actually buy and become a customer. And casual visitors, just browsing your store, catalog, or advertisements are similarly anonymous. Web-based e-commerce is unique in that it can capture all the pre-sales activity of prospecting potential customers as well as browsing visitors, greatly enhancing the enterprise's overall market knowledge, and permitting much more sophisticated customer acquisition and retention strategies. As mentioned earlier, in an e-commerce environment we lump visitors, prospects, targets, and customers into the general category of users.

Let's start with the data warehouse schema of a forward-looking brick-and-mortar retailer and see how it changes in an electronic commerce environment. See Figure 11.
 
 

Figure 11: Forward-looking brick-and-mortar retail data warehousing

The e-commerce data warehouse site activity fact table records much more than just sales activity.
The traditional brick-and-mortar Sales Fact Table becomes the User Activity Fact Table in the e-commerce environment, as shown in Figure 12. While actual sales transaction information is all that is typically known in the brick-and-mortar world, e-commerce sites can record all site user activity, including that of prospecting buyers, targeted users and casual visitors.

Figure 12: The traditional sales fact table becomes the user activity fact table in the e-commerce environment

On the Web, the pre-sales activity of actual customers can be recorded in minute detail. Facts that can be recorded in a User Activity Fact Table include activity source, time spent on the activity, activity cost, and activity revenue. For example, an activity source might be a parent Web page URL, or the TCP/IP address of a site user coming into the site. Time spent is the elapsed time spent on a particular site Web page or frame. The site activity cost is the dollar cost to the enterprise for the activity on the page or frame, and the activity revenue is the revenue gained from the site activity, both of which can be any number greater than or equal to zero. Each of these site activity facts has a composite key from the associated e-commerce data warehouse dimension tables, which are explained below. The voluminous clickstream detail creates an explosion of fact table information that makes scalable data warehouse environments an absolute necessity.

e-commerce site users are more than customers.
The leading-edge Customer Dimension from the brick-and-mortar world becomes the User Dimension in an e-commerce environment, as shown in Figure 13. External e-commerce site users are visitors that can be any one of customers, prospecting potential buyers, or casual visitors, and all their site activity is easily recorded by Web logging mechanisms. But it is important to note that e-commerce site users do not necessarily have to be external to the enterprise.

If customer service and call center personnel use a Web-based system, then customer service call center site activity can be recorded in the same data warehouse schema that is used for external clickstream activity. This realization is an important breakthrough, because it links all user contact activity in a single data store, whether the method of contact is via the Web or via the telephone or e-mail. All electronic user activity is recorded, regardless of media, in the unified clickstream/callstream data warehouse. And the knowledge gained from the full spectrum of user activity stored in this unified analytical model gives significant competitive market and customer knowledge advantages to e-commerce enterprises.

Figure 13: The customer dimension becomes the user dimension in an e-commerce environment

In the press and analyst reports, much is made about the difficulty of identifying a visitor to a Web site, because, at a minimum, all that is known about a visitor is his originating IP address and nothing else. While this is a problem, it pales when compared to that of the traditional brick-and-mortar retailer, who typically has no idea who visited a store (walked in and then out of his store), what they did while there, or which potential customers scanned and silently rejected expensive print or media ad-campaigns or never read them at all. In contrast, the e-commerce entity can capture the details of all client visits and Web ad-induced click-throughs, and while they may not know the client's exact identity, they at least know that he got to the site and what he did there. The analysis of his behavior is significant, even if his actual identity is unknown. This is a significant increase in customer/prospect/visitor knowledge, and it gives electronic enterprises a significant competitive advantage over brick-and-mortar competitors.

Geography gains fine detail on the Web.
The traditional brick-and-mortar, physically-oriented geography dimension goes virtual in e-commerce, and the result is three new location-oriented dimensions. See Figure 14.

Figure 14: The traditional physically-oriented geography dimension goes virtual in e-commerce

 

Time goes individual on the Web.
Because e-commerce enterprises have users that can be located across the globe, the traditional Time Dimension splits into the financially-oriented Fiscal Time Dimension and a physical-geography-specific User Time Dimension, as shown in Figure 15.

Figure 15: The traditional time dimension splits into the financially-oriented fiscal time dimension and a physical-geography-specific user time dimension

Fiscal Time defines the fiscal year of the e-commerce enterprise, but User Time defines the user-oriented time of day characteristics like morning, afternoon, evening, the season of the year, etc.

e-commerce seasonality is non-intuitive without the User Time dimension. For example, Northern hemisphere users can be in User Time summer while Southern hemisphere users are simultaneously in the User Time winter season.

Content, not just products, on the Web.
The traditional brick-and-mortar Product Dimension changes dramatically into the content and activity dimensions in an e-commerce environment, as shown in Figure 16. An e-commerce site's business is defined by its content and that may include products for sale, but is rarely exclusively so. Examples of other e-commerce offerings include interest-group information, downloads, internal advertising, external advertising (click-throughs), customer service, etc. All of this is described in the Content Dimension.

Figure 16: The product dimension changes into the content and activity dimensions in e-commerce

Coupled with content is the notion of Activity, which indicates what someone did in response to the content. Examples of activities include click-downs to related pages, click-throughs to external advertising, information downloads, purchases, service calls, help, etc. All of this is captured in the Activity Dimension.

Advertising goes external on the Web.
The traditional brick-and-mortar Promotion Dimension expands its focus beyond internally-focused advertising and sales promotions to include externally-focused promotions on the Web, as shown in Figure 17. Most e-commerce Web sites get revenue not only from sales, but also from external promotional tie-ins, some almost exclusively so. While brick-and-mortar retailers advertise brand name merchandise for sale and often get compensated for those efforts to build an external brand, e-commerce enterprises can have external advertising relationships which extend far beyond those found in traditional commerce.

For example, electronic ad servers serve up advertisements that are targeted at a user's behavior profile and, in return, the site receives click-through revenue based on user ad-click activity. The increased focus on external advertising revenue in e-commerce, and the different business goals of internal and external promotions, cause the single traditional pro-motion dimension to split into the Internal and External Promotion Dimensions.

Figure 17: The promotion dimensions expands to include externally-focused promotions on the Web

Another distinguishing characteristic is that Web-based promotions are much more finely targetable than with traditional brick-and-mortar retailers. Also, customer-acquisition media, like Internet interest sites and chat rooms, are more finely targeted with richer media than was previously possible. This means, for example, that a mountain climbing gear retailer may prosper on the Web, where it might have to be part of a sporting goods chain to survive in the brick-and-mortar world. This ability to more finely target a wider geography and, hence, larger interest group is one of the key drivers behind the profusion of business-to-consumer electronic commerce enterprises.

A complete clickstream/callstream data warehouse schema.
The end-result is the clickstream/callstream data warehouse schema shown in Figure 18. As you can see, the Web changes everything.
 
 

Figure 18: e-commerce clickstream/callstream data warehouse schema
 
 

Problem three: alignment of information technology vendor and e-commerce enterprise business goals
e-commerce solutions have grown-up in a modern, open systems technology environment. While open systems solutions have many advantages, they also can create fundamental business goal alignment problems between the information technology product vendors and e-commerce enterprise that can thwart effective e-commerce information technology solutions. As mentioned earlier, e-commerce enterprise employees have a natural, job-performance-driven tendency to try to become e-commerce knowledge and management Supermen, attempting to orchestrate a hodge-podge of information technology vendors, including hardware, software and services, none of whom have any real stake in the success of e-commerce enterprise. This situation is illustrated in Figure 19.

Figure 19: Multi-vendor solution

While a low cost, best-of-breed e-commerce solution, integrated by in-house personnel may seem attractive, there are several pitfalls. Unfortunately, the individual suppliers of information technology have no real stake in the success of the total e-commerce solution. They care only about their piece of it, and their economic incentive is focused primarily on the initial sale, not on long-term success of the e-commerce enterprise. The myriad of best-of-breed point-product solution vendors creates the need for deep in-house integration expertise that is vulnerable to employee knowledge, mobility and reorganization issues. Furthermore, long-term integration costs are borne solely by the e-commerce organization, and this can be an increasingly burdensome issue as time progresses, and the complexity of integration issues grow.

A natural reaction to these problems is to outsource at least a portion of the solution to a solution integrator, as shown in Figure 20.

Figure 20: Outsourced solution

While outsourcing creates an attractive short-term management solution to the problems of an IT-lead best-of-breed e-commerce solution integration, the motivations of the outsourcer and the e-commerce enterprise are still not properly aligned. These problems surface in a number of subtle ways.

In an outsourced solution, the integration vendor is the primary provider of the solution, and they are likely the sole source of the intellectual capital for the solution. Because the contracting e-commerce enterprise is relieved of much of the responsibility for the creation of internal solution expertise, the resulting outsourced solution is limited by the knowledge, business relationships, and integration expertise of the integration vendor. These limitations can lead to long-term solution issues that cannot be easily solved by the contracting enterprise because the expertise required to do so is non-existent by definition梚t was outsourced.

This is bad enough, but the motivations of the integration vendor are rarely aligned with e-commerce enterprise business success. Most integration contracts are based on internally-focused time and materials pricing, which has nothing to do with e-commerce business success. Furthermore, there is an implicit "forever" term to outsourcing contracts, meaning that the e-commerce enterprise will pay for this external solution expertise forever, because they are abdicating the development of the same in-house solution capabilities. When coupled with the time and materials nature of integrator pricing, this can lead to an incentive for the outsourcer to never solve any fundamental business problems, because if they did, it would only reduce their revenue.

An additional worry is the practical inability to write umbrella outsourcing contracts that address all the potential information technology modifications required to support unanticipated changes in business conditions. In an Internet environment, the contract-induced inability to rapidly adjust to changing business conditions can be fatal to the enterprise.

These insidious issues associated with totally outsourced solutions lead to a middle ground. While this situation is not without problems, it better aligns business goals of the e-commerce enterprise and integration solution vendor. See Figure 21. In order to solve the implied "forever" solution term with the integration vendors, the outsourcing contract must be fixed-term and non-renewable, and it must mandate sign-off criteria and specify knowledge transfer to responsible individuals in the e-commerce information technology organization. While this type of arrangement can still lead to higher initial solution costs and a solution limited by the knowledge of the integrator, the business drivers behind the motivations of both parties are much more aligned to the success of the e-commerce enterprise. The integration vendor takes time-limited and performance-goal-specific responsibility for a successful solution, and the resultant knowledge transfer causes the contracting e-commerce information technology organization to learn the skills required for long-term stewardship of the solution.

Figure 21: Fixed-term and non-renewable outsourcing solution

Why IBM NUMA-Q® for your e-commerce solution?
The answer starts by pulling apart the technical underpinnings of an e-commerce solution, as shown in Figure 22.

Figure 22: IBM NUMA-Q e-commerce system solutions

IBM NUMA-Q has the most scalable, available, and easy-to-manage open systems solutions:
 

IBM NUMA-Q brings a scalable, highly available, mission-critical solution focus to the e-commerce business environment:
 
 

IBM NUMA-Q has the design, implementation, and solution integration expertise required for all the critical back-end systems that are the key to the long-term success of an e-commerce enterprise:
 
 

IBM provides, as an explicit deliverable, knowledge transfer:
 
 

IBM provides the fastest route to a solution:
 
 

&