In a classic online book-store example, book details, prices and shop categories are all generic. They do not depend on any particular user. Browsing the catalogue should produce the same results for any two users at the same moment. Actions like buying a book or reviewing orders and transactions are user specific and they cannot be shared between different users. Why is it so important to split these two streams right from the start? Because generic and user-specific data flows are affected by completely different caching and partitioning constraints.

Different constraints

Keeping session footprint low, caching and partitioning have been tried and tested as best practices for scaling web systems. If there is only a single data stream, because of different constraints, these practices will be hard to implement and even harder to improve later.

For example, most generic data flow is completely stateless, but user-specific actions are often stateful. If these two flows are clearly separated, then we can process generic actions with stateless services and servers. Stateless services are much easier to manage and scale than stateful services, since they can be easily replaced without any effect on the system operation. You can just stack more cheap servers and throw requests at them using round-robin or simple load balancing when you need more power. Stateful services cannot be scaled so easy — they might rely on resource locking and load balancers have to send all requests for a single session to the same server. If a stateful server dies, that has a visible effect on the system operation, so these services have to be much more resilient than stateless ones.

Generic data can be cached and shared between users and even servers. User-specific data like account information or transaction statuses should never be cached. Similarly, there is most likely no authentication required to get the generic data, but in most cases authentication rules apply to user-specific data. For example, anyone can view book details, but a user has to be logged in to view an order status (and the requested order has to belong to that particular user). If the two data streams are mixed, then any caching mechanism will have to analyse and understand the context of requests, and decide what should be cached on case-by-case basis. If the two streams are split, the decision is easy: everything generic is retrieved from the cache, everything user-specific does not even try to use the cache. No error-prone analytics involved. The generic data flow also does not have to suffer from the authentication overhead — you can save a lot of processor power by skipping authentication when it is not needed. Remember the Golden rule of web caching? We can cache generic data into static files and avoid any dynamic processing on the web servers, getting much better performance. On e-commerce systems, user-specific pages typically have to go through HTTPS, which adds even more overhead.

Because it does not matter where the generic data flow comes from, it is also much easier to partition. Once the system grows over the capacity of a single database, we can just take the generic content out and split it into several databases. Splitting account data and transactions is never that easy, because those details are bound by a single context and often constrained by business rules and validations.

How to split the flow

In the classic two-tier architecture, where the web site plugs directly into the database, data flow splitting starts with using a different data source for those two streams. Even if, on the start, everything sits in the same database, use different connection pools to retrieve user-specific and generic data. The generic data flow does not even have to run in transactions, which can make a big difference with some databases. If an ORM mapping library is used, then caching should be turned on for this stream. User-specific data flow, as a general rule of thumb, should be transactional and should never be cached. It is a good idea to hide any user-specific tables from the generic data flow using database user privileges, so that any attempts to mix the two streams get caught quickly. Internal web site services should be designed to clearly fall into one of those two categories, and not mix methods that retrieve generic data and perform user-specific functions. This is all a preparation for the day when the two-tier system needs to be split into more levels. With a clear separation of responsibilities, this divorce should not be painful at all.

In three tier architectures, I like to split the middleware straight from the start into user-specific and generic servers. Web servers sit on top, get the generic data from the first middleware group and process transactions using the second middleware group. Generic data-flow servers can be clustered and scaled easily, and any load balancing system will work right out of the box. They can be restarted, taken out of the cluster or put back in without any effect on the rest of the system. Transparent caching can be applied to those servers easily. User-specific servers, on the other hand, are much more tricky in all those aspects and should absolutely never be transparently cached. This split is a preparation for further scaling and caching, since generic data servers can be split regionally, put under several layers of cache servers, divided vertically by product range or type. The functionality on user-specific servers is focused and isolated, so that we will have less to focus

The primary aim of caching is to speed up processing and reduce load on critical resources. In case of web sites, the most critical resource is probably going to be the database – so dataset caching is a common solution to speed up the web. It is relatively easy to implement, and it can be introduced almost transparently into an existing system. But that is just painting the corpse. First of all, dataset caching requires the same source data to be re-processed over and over, and takes up unnecessary CPU cycles. Also, a lot of data from recordsets is often not displayed directly, so dataset caching can lead to a lot of wasted memory – in fact, I’ve seen sites where this approach caused serious problems by fragmenting memory under heavy load and leaving the application without enough memory to run.

Although the web site will, no doubt, benefit significantly even from a dataset cache, it will scale much more and better if the cached content is closer to the final product. How much closer, that depends on the application. The golden rule of web caching is: For the caching to be most effective, cache as close to the final product as possible.

Ideally, cache content into static files

The best option for caching, if possible, is to use static files on the disk and allow web servers to publish those files directly. All web servers will process static files very efficiently, and that will also leave more resources for stuff that needs to be processed dynamically. Content on the disk does not take up valuable memory space available for the web application, and it absolutely minimises CPU requirements per request.

Here are the results of a stress test I recently did on a fairly good web server machine running IIS6 over a gigabit network. I measured the number of requests per second during a two-minute stress load for serving the a couple of files using several techniques.

• Static – file served directly by IIS from disk
• ASP-SSI – file included using #include SSI directive in an ASP file (turning on the SSI engine)
• ASP-Include – file included using #include SSI directive in an ASP file but with an ASP statement if (true) around the include (so turning on the ASP engine as well)
• ASP-Cache – file read from disk using FSO, but cached in an Application object for up to 10 seconds.
• ASP-Nocache – file read from disk on every request using FSO
• ASPX-SSI – file included using #include SSI directive in an ASPX file
• ASPX-Include – file included using #include SSI directive in an ASPX file, with an additional C# if (true) block around the include
• ASPX-Cache – file read from disk but cached using ASP.NET page caching for 10 seconds
• ASPX-NoCache – file read from disk on each request, page caching turned off

The difference between “static” and “aspx-cache” is the pure overhead of using the ASP.NET engine. For a 4KB file, we get a 38% increase of performance from a file based cache over even the ASP.NET page caching mechanism. Compared to a simple cache based on the Application object in ASP, the performance increase is 150%. And this is just for managing completely static content. A typical web application would pull the content from a database or format it, leading to much bigger differences. Also interesting is the difference between ASP-Include and ASP-SSI, effectively the cost of having turning on the ASP engine (having if(true) statement in ASP code). The good news is that difference between those two cases in ASP.NET is negligible.

With larger files we hit the bandwidth bottleneck first, so the benefits of a file based cache are not that visible. However, with the growing number of Ajax-based sites, pages are getting split into smaller independent requests, and so the overhead becomes quite noticeable. Although ASP.NET page caching is a great utility given that it almost requires no work to implement, file based cache can also be split across the server farm, allowing multiple machines to use the same content.

Caching into static files also brings the benefit of automatic support for last modification timestamps. For browsers with HTTP 1.1 support, if the same files are downloaded often (live Ajax updates), the server can actually reply just with the “304 Not modified” header, without sending any file content at all. Cache that works directly from disk files also allows us to use lightweight HTTP servers such as LightHTTPD to get some extra performance from the same hardware.

Not just for pre-publishing

The file based cache is often used for pre-publishing content, but there is a simple trick to use this technique for on-demand publishing, even for content that must be generated often on the fly. Most web servers I have worked with, including IIS and Apache, will allow us to override the 404 “Not found” handler. This can be used to implement the cache-miss scenario: when users request a file that has not yet been cached, IIS will not find it on the disk and will call the 404 handler; we can then generate the file using dynamic processing (ASP/ASP.NET) and store it to the disk for the next request before sending the content back to the client. The same technique can be used with Apache and PHP.

An important note for this technique is that requests must be mapped completely to the file path of the URL, not to GET parameters. So, for example, instead of using http://myserver/search.aspx?query=fitnesse, we would have to use something like http://myserver/search/fitnesse.query. Then a 404 handler would be set up on the search folder to perform a search based on the active URL path and store it into the fitnesse.query file. You can generally chose any extension you want, but avoid ASP/ASPX and other standard extensions, to prevent IIS from turing on ASP/ASP.NET processing when serving those files. HTML and TXT extensions are also not a good choice. IE uses “smart” caching by default, so HTML files are downloaded just once per page. If you fire a background AJAX request twice for a HTML file, only the first will actually go to the server. TXT and HTML files may also be cached by transparent proxy systems, so it’s best to avoid using those extensions.

Image credits: Akis Kolokotronis/SXC

Enabling the cache

To turn on caching for a function, add RESULT_CACHE immediately before IS or AS in the function declaration. There is an optional RELIES_ON clause which can specify a database table or column dependency. If a dependency is specified, any change to the underlying object invalidates cached results. So, for example, we can write a function that does some on-the-fly statistics depending on a configuration parameter. If the parameter table changes, function results are automatically deleted from the cache. This mechanism is intended to replace all in-house caching systems developed over the years using triggers and PL/SQL package variables.

A new DBMS package dbms_result_cache can be used to get statistics and manipulate cache directly. Two most interesting methods in that package are memory_report and FLUSH. Flush clears the cache, and memory_report prints a nice memory usage report to serveroutput.

A practical example

So let’s try it out. We’ll do some statistics on a table, and run the query a couple of times without and with caching. I typically run tests like these on sys.all_objects, but for some reason result caching is disabled on SYS structures. For the test, we’ll first copy ALL_OBJECTS into a table in some user schema. You’ll probably need the DBA role to run this script because of the memory flushing, but result caching is available to normal user roles.

spool c:\log.txt 

drop table BIGTABLE;

create table BIGTABLE as
select * from all_objects;

SET SERVEROUTPUT ON

exec dbms_result_cache.flush;

alter system flush shared_pool
/

exec dbms_result_cache.memory_report;

create or replace function COUNT_OBJECTS(pOBJECT_TYPE varchar2) return number as
lCount number;
begin
	select count(*) into lCOUNT from BIGTABLE where object_type=pOBJECT_TYPE;
	return lCOUNT;
end;
/

create or replace function COUNT_OBJECTS2(pOBJECT_TYPE varchar2) return number
RESULT_CACHE relies_on (BIGTABLE) 
as
lCount number;
begin
	select count(*) into lCOUNT from BIGTABLE where object_type=pOBJECT_TYPE;
	return lCOUNT;
end;
/

set timing on;

set autotrace traceonly;

select object_type, COUNT_OBJECTS(object_type) from BIGTABLE group by object_type;

select object_type, COUNT_OBJECTS(object_type) from BIGTABLE group by object_type;

select object_type, COUNT_OBJECTS2(object_type) from BIGTABLE group by object_type;

exec dbms_result_cache.memory_report;

select object_type, COUNT_OBJECTS2(object_type) from BIGTABLE group by object_type;

exec dbms_result_cache.memory_report;

spool off;

Methods COUNT_OBJECTS and COUNT_OBJECTS2 are absolutely the same, except for the result caching. When the script is executed, we see the difference in the trace and memory report. The first execution, without caching, comes to this:

36 rows selected.

Elapsed: 00:00:00.42

Execution Plan
----------------------------------------------------------
Plan hash value: 2406321947                                                     

-------------------------------------------------------------------------------
| Id  | Operation          | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |          | 63870 |   686K|   284   (2)| 00:00:04 |
|   1 |  HASH GROUP BY     |          | 63870 |   686K|   284   (2)| 00:00:04 |
|   2 |   TABLE ACCESS FULL| BIGTABLE | 63870 |   686K|   281   (1)| 00:00:04 |
------------------------------------------------------------------------------- 

Note
-----
   - dynamic sampling used for this statement                                   

Statistics
----------------------------------------------------------
        248  recursive calls
          0  db block gets
      37423  consistent gets
       1006  physical reads
          0  redo size
       1179  bytes sent via SQL*Net to client
        356  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
          2  sorts (memory)
          0  sorts (disk)
         36  rows processed                                                     

The second one, calling the same function but without caching, leads to this:

36 rows selected.

Elapsed: 00:00:00.37

Execution Plan
----------------------------------------------------------
Plan hash value: 2406321947                                                     

-------------------------------------------------------------------------------
| Id  | Operation          | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |          | 63870 |   686K|   284   (2)| 00:00:04 |
|   1 |  HASH GROUP BY     |          | 63870 |   686K|   284   (2)| 00:00:04 |
|   2 |   TABLE ACCESS FULL| BIGTABLE | 63870 |   686K|   281   (1)| 00:00:04 |
------------------------------------------------------------------------------- 

Note
-----
   - dynamic sampling used for this statement                                   

Statistics
----------------------------------------------------------
         38  recursive calls
          0  db block gets
      37222  consistent gets
          0  physical reads
          0  redo size
       1179  bytes sent via SQL*Net to client
        356  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
         36  rows processed                                                     

The number of physical reads and recursive calls is much lower, because of internal caching mechanisms. Execution time drops from 42 to 37 ms. Now, the third execution with caching turned on:

36 rows selected.

Elapsed: 00:00:00.37

Execution Plan
----------------------------------------------------------
Plan hash value: 2406321947                                                     

-------------------------------------------------------------------------------
| Id  | Operation          | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |          | 63870 |   686K|   284   (2)| 00:00:04 |
|   1 |  HASH GROUP BY     |          | 63870 |   686K|   284   (2)| 00:00:04 |
|   2 |   TABLE ACCESS FULL| BIGTABLE | 63870 |   686K|   281   (1)| 00:00:04 |
------------------------------------------------------------------------------- 

Note
-----
   - dynamic sampling used for this statement                                   

Statistics
----------------------------------------------------------
         58  recursive calls
          0  db block gets
      37310  consistent gets
          0  physical reads
          0  redo size
       1180  bytes sent via SQL*Net to client
        356  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
         36  rows processed

Number of recursive calls and physical reads is still lower than in the first case, because table data is loaded into the memory. Execution took about 37 ms, more or less the same as in the second run. Consistent gets show that we are still processing the whole table, because they are basically the same as in the first run.

The memory_report procedure after this call shows that some data is in the cache:

R e s u l t   C a c h e   M e m o r y   R e p o r t
[Parameters]
Block Size          = 1K bytes
Maximum Cache Size  = 4064K bytes (4064 blocks)
Maximum Result Size = 203K bytes (203 blocks)
[Memory]
Total Memory = 202184 bytes [0.057% of the Shared Pool]
... Fixed Memory = 5296 bytes [0.002% of the Shared Pool]
... Dynamic Memory = 196888 bytes [0.056% of the Shared Pool]
....... Overhead = 131352 bytes
....... Cache Memory = 64K bytes (64 blocks)
........... Unused Memory = 26 blocks
........... Used Memory = 38 blocks
............... Dependencies = 2 blocks (2 count)
............... Results = 36 blocks
................... PLSQL   = 36 blocks (36 count)
PL/SQL procedure successfully completed.

Now the last execution – repeating the function with caching turned on:

36 rows selected.

Elapsed: 00:00:00.01

Execution Plan
----------------------------------------------------------
Plan hash value: 2406321947                                                     

-------------------------------------------------------------------------------
| Id  | Operation          | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |          | 63870 |   686K|   284   (2)| 00:00:04 |
|   1 |  HASH GROUP BY     |          | 63870 |   686K|   284   (2)| 00:00:04 |
|   2 |   TABLE ACCESS FULL| BIGTABLE | 63870 |   686K|   281   (1)| 00:00:04 |
------------------------------------------------------------------------------- 

Note
-----
   - dynamic sampling used for this statement                                   

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
       1006  consistent gets
          0  physical reads
          0  redo size
       1180  bytes sent via SQL*Net to client
        356  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
         36  rows processed                                                     

So between third and fourth call, number of consistent gets dropped significantly. There were no recursive calls and no function executions. All values were read straight from the cache. Notice how the execution time dropped from 36 ms to 1 ms.

The cache is in the system global area (SGA), so cached values are available across database sessions. So, to conclude, adding RESULT_CACHE is an easy way to optimise dynamic statistics and parametrisation. Here are some links for the end:

• Improving Application Performance with Result Cache article from OTN.
• DMBS_RESULT_CACHE API docs

• Result caching syntax docs