Javasmith

Mockito is just awesomeness

2012-01-28T22:41:00.000-08:00

Last week I ventured into making some enhancements into a project that is completely unknown to me. I see unit tests as my writ of passage before I can checkin any code into a project new to me --not to say I should ever checkin code without tests, but I think you all know what I mean.

Writing tests for an unknown code is either difficult or tiresome; difficult, if I decide to understand large chucks of the project before writing a lick of code; tiresome, if I chose to only understand the module that I am changing and just mock the rest of classes when unit testing. In reality, It should be a bit of both.

In the past I made lots and lots of mock objects by making classes that implement a certain interface or try to instantiate a class with lots of dependency, when really, I only care about one or two methods in the class. It's a tiresome, verbose, and just plain an ugly thing to do for writing a unit test.

Fortunately, I came across Mockito with allows you to mock and stub classes effortlessly.

Take the StatsManager class that has a genderCount() method.

public class StatsManager {

    private JDBCTemplate template;

public long genderCount(Gender gender) {
   //perform some business logic

...
}

public void setJDBCTemplate(JDBCTemplate template) {
this.template = template;
}

...

}

Say I want to make a change to the genderCount() method and I need an unit test to validate my change.

Before Mockito, I would need to instantiate an instance of JDBCTemplate. It happens that JDBCTemplate needs an javax.sql.DataSource instance to work. I would then try to make a mock class of this interface, but then it turns out that javax.sql.DataSource has a dependency javax.sql.Connection so I have to mock that class and world goes round and round... but I never finishing mocking!

What is it that I really need in order to test this code? It's the logic in genderCount() that I want to test, and don't want to concern myself with all these other setup classes/interfaces. Infact, I know that queryAsList(String sql) is the only method called on the JDBCTemplate class in StatsManager.genderCount() . All I want is a certain List of results when genderCount() is called.

This is exactly what Mockito give me! Here is an example

public class StatsManagerTest {

@Test
public void testGenderCount() {

StatsManager manager = new StatsManager();

JDBCTemplate template = mock(JDBCTemplate.class)

List results = new ArrayList();
//initialize list.
...

when(template.queryAsList(anyString())).thenReturns(results);

long maleCount = template.genderCount(Gender.MALE);
//Assert a bunch of stuff
...

long femaleCount = template.genderCount(Gender.FEMALE);
//Assert a bunch of stuff
...

The key thing to notice here is the when() and thenReturns() pair. What this is doing is that when queryAsList() is called regardless of what string is passed as a parameter, return this fixed list of results.
That's it! My unit test is concise/readable, free from a bunch of setup code. I can always setup a different list of results to test against in my unit test.

It makes writing unit tests fun versus a necessary evil.

Back from Java haitus

2012-01-28T21:02:00.000-08:00

It's been a whole 10 months since I've written any sensible Java code. Been developing a LAMP stack (we'll not exactly; lighttpd instead of apache). Lots of PHP, Javascript/JQuery, HTML, CSS. Let's just say I've been itching for the type-safety.

This coming up in the next couple of months: Hadoop, Hive, Cascading, Spring, Membase and maybe even some Scala and Akka! Stay tuned...

BigMemory is out! Garbage Collection RIP

2010-11-09T13:18:00.000-08:00

I'm sure like many of you, I tend to browse DZone for interesting articles/blog posts. As a guy who works on Java server-side software, articles related Garbage Collection always catches my eye. Today while browsing on DZone I came across this article on Garbage Collection tuning.

Halfway through reading the article I realized the irony: I'm reading a pretty well written article on how to tune your Java Garbage Collector on the same day Terracotta announced GA for our BigMemory product. As I spelled out here and here BigMemory makes the need for tuning unnecessarily. And if your talking about Java Heaps large; GC Tuning? Forgetaboutit!

I know as a developer I would rather work on making my product better, faster and more concurrent then dread that GC gremlin showing up pausing my application and killing the buzz. If you do want to make YOUR application better and not let GC get in the way to that, check out the BigMemory product here.

Articles on Garbage Collection just got a whole lot less interesting.

Ehcache BigMemory: Simple High Availability, Even Simpler

2010-10-26T22:00:00.000-07:00

My collegue Jason @ Terracotta did a nice post on using Ehcache with JRuby; which led to a discussion on a long list of features we implemented for Enterprise Ehcache (check out the discussion thread on Jason's Blog).

Adding to Jason's feature list, I would like to discuss HA (High Availability) in Ehcache and explain why our BigMemory product makes tuning HA even simpler. Lets review how we do HA in Enterprise Ehcache. In Enterprise Ehcache, clients going down is no big deal since the data is present on the servers as well. Our HA focuses on protecting our servers. In Enterprise Ehcache you can define one or many server groups. Each group consists of a cluster of servers. The cluster has to decide which node is going to be the active server. This is decided by having an election where a node is selected to be the active server. The rest of the nodes in the cluster are waiting in passive standby ready to take over if the active node fails.

In order to actually detect when a failover needs to happen, we wrote a configurable HC (Health checker). Our HC detects errors that won't show up as a normal network disconnect or failure, such as a network cable being pulled. Because Enterprise Ehcache is written in Java, we also had to deal with long GCs. So we designed our HC to detect long GCs as well.

Based on your use case, you may want to change the HC settings depending on what your tolerance is for network disruption and long GCs. Before you go about changing your settings, you might want to check out these files:

 $TERRACOTTA_KIT/platform/config-samples/tc-config-healthchecker-aggressive.xml
$TERRACOTTA_KIT/platform/config-samples/tc-config-healthchecker-aggressive.xmltc-config-healthchecker-development.xml
$TERRACOTTA_KIT/platform/config-samples/tc-config-healthchecker-aggressive.xmltc-config-healthchecker-production.xml

Depending on what you're doing and what your requirements are, picking one of the settings above should be suffice.

Now let's discuss these properties:

 l2.healthcheck.l2.ping.idletime=3000
l2.healthcheck.l2.ping.interval=1000
l2.healthcheck.l2.ping.probes=2
l2.healthcheck.l2.socketConnectTimeout=5
l2.healthcheck.l2.socketConnectCount=2

Above are the properties you have to work with for HC. The HC starts off using the ping.idletime. This is the maximum amount of time that can elapse between the last time data was received from the corresponding node. In this case the idletime is 3000 milliseconds, after which the HC notes "Hey, didn't receive any data from the corresponding node, I should check on that node."

To check on the health of the node, it tries to ping the node in intervals, in which the interval length is defined by ping.interval. You can push this number down to get more granularity. If the corresponding node doesn't respond within the ping.interval then it either tries to probe again because the ping.probes countdown hasn't completed, or it checks socketConnectCount and see if its allowed to make any more socket connections. If not, it declares the corresponding node DEAD.

In the example above, since the socketConnectCount is set to 2, it will try to make another socket connection. If it cannot make the socket connection within (socketConnectTimeout * pingInterval) ms, then it will declare the node DEAD. In our example, the interval length 5000 ms. Once it established a connection it will repeat the ping probe cycle again.

The maximum time it will take the HC to detect a network disruption is ( ping.idletime + (ping.probes * ping.interval) + (socketConnectTimeout * ping.interval) ) ms. If the problem is longGC, then the connection will happen, but the pings won't receive a response. The maximum time HC takes to detect a long GC is ( socketConnectCount * ( ping.idletime + (ping.probes * ping.interval) + (socketConnectTimeout * ping.interval) ) ) ms.

If you have a short tolerance for network disruption, but your ok with having lengthy long GCs, then you can decrease the ping.idletime and increase the socketConnectCount; you tune based on your tolerances. Here's some detailed documentation on the HA settings.

With BigMemory in our server FORGET ALL THAT IS WRITTEN ABOVE.

Our HC has all these different properties because we had to be tolerant of Long GCs. When a node is in long GC, it will make a socket connection but not be able to complete the ping probe cycle. but now with BigMemory you probably don't ever need to change these settings, unless you have people tripping over your network cables.

Unlike Long GCs, network disruptions is something you probably know about and its easier to guess what that tolerance should be. Not having to tune for long GCs makes HC configuration simple. You only need to tune for YOUR own environment (i.e. crappy network, or clumsy workers) and not for something that is specific to Java (Long GCs).

Imagine what it can do for you. Check out our beta here.

BigMemory: Followup Q and A

2010-10-17T10:44:00.000-07:00

I got quite a few responses to my post on BigMemory in the Terracotta Server. It seems like people are quite confused on what it actually is.

Here's some answers to a few questions I received:

1. Why can't they (Terracotta) put garbage collector on another cpu core and gain performance?

I think there is a misunderstanding about the cost of Garbage Collection. The Full GC pause (which is when all application threads are paused) is what the GC problem in Java is all about. It is tolerable when your Heap is 1-2 GB. But anything beyond that you get 4,5,8 seconds GC pauses. Besides, if you don't run ParallelGC then it will use one core anyway. But you DO want to have your garbage collector using all the cores so it will complete faster and have less pauses.

2. (In References to the question above) Then put it on another thread and how about pausing one thread at a time ?

Again this is not possible AFAIK to do with the Sun/Oracle JVM. Also, Full GC Pauses are a necessary evil for the GC algorithm they are using. Even if this was possible, it would not solve the problem of unpredictability.

3. I can't believe there are no GC pauses ... or you guys might have made memory management solution like an OS in java.

The idea of have direct memory allocation in Java is no big secret. There is an -XX:MaxDirectMemorySize flag to tell the JVM how much direct memory to allocate. The value add of Terracotta is to use this direct memory space in a way that is fast and does got fragment.

4. Using direct memory allocated by the JVM is useless because it is so much slower then the Heap.

Access to direct memory is NOT slower than Heap. There are two things that contribute to the perceived slowness of direct memory. Serializing and deserializing data to and from direct memory; and allocating and cleaning up direct memory buffers. At Terracotta we solved the direct memory and cleanup problem. On the Terracotta Server we don't pay for the serialization/deserialization cost. On Enterprise Ehcache (unclustered) we do pay a serialization/deserialization cost, but compare this CPU cost to having to deal with Full GC Pauses on the Heap. The tradeoff is well worth it. Besides BigMemory using the Heap as part tier storage strategy; Heap to OffHeap to Disk. It's an age old principle in computer science (think Virtual Memory). We avoid the serialization/deserialization cost for frequently used objects by having those in Heap, then having a big part of your cache on OffHeap to avoid long FullGC and the rest spilling over to disk.

For the additional CPU cost what you get in return is predictable latency and speed with all the memory your Java process desires. Find an app where you do see Full GC pauses and checkout the beta to see for yourself.

OmmWriter Plug

2010-10-14T00:22:00.000-07:00

Just wanted to plug this fantastic software for the Mac. OmmWriter. Writing is a painful excercise for me since I get distracted pretty easily. With so many application like IDEs, chat, mail, brower, twitter etc. open its really hard to concentrate. When I try to write about any topic, I always have a tendency to fact check every sentence, which takes me on reading tangents. OmmWriter runs in full-screen mode and removes any sort of notification. I can write with purpose and bang out a blog post really quickly. Give these guys a try, their version 1 is free and the newest one is a paid version.

Terracotta BigMemory: A tale of eating our own code

2010-10-13T23:51:00.001-07:00

Terracotta (The company I work for) recently came out with a beta release for their BigMemory product. Our claim is that if you use Ehcache BigMemory then those GC problems go away. If you don't know what Ehcache or Terracotta is you can find out here.

This is far from just a claim, we didn't simply didn't write BigMemory, ran a few tests, and release the Beta. We took BigMemory and freed our own Terracotta Server for GC constraints that most Java server product have.

So what exactly did we do? In Terracotta we keep a representation of distributed objects in your server. For our clustered Ehcache product we keep a representation of cache segments and cache entries on the server. As the cache grows, we need to keep track of the keys associated with the cache segment, as well as have more cache entries representations on the server. The Terracotta Server has the ability to flush cache entries to disk when we detect memory pressure on our server. and fault in objects when cache entries are needed.

To accommodate the keys on the server, you had two options. One is the increase heap so that more keys could fit on the server. The second is to add additional Terracotta Servers to the cluster so that the segments (and their keys) are distributed to among many servers.

Adding more heap threw us into the classic GC problem. Once you start getting into heap sizes of 5-6GB, you start seeing 5-8 second GC pauses. Adding stripes solves the problem and worked for our customers as well.

But we found a pattern emerging. Customers were purchasing huge boxes (I'm going regret saying huge a few months from now ), 32 GB ram and 16 cores of processing power. They were like "We want to run your servers on this thing." And they did what everyone does for Java Servers, they basically run multiple JVMs (if possible) on the same box and deal with the added complexity and unpredicatability and probably says a prayer or two. I'm sure you all understand what I mean by complexity, but what do I mean by unpredictability. Lets say you run your servers with small heaps and your getting a 2 second GC, but will so many JVMs, how do you know your GCs will not be staggered. Meaning if you had 16 processes running and they GCed one after the other. That is a 32 second pause. Get the picture?

We figured, there must be a better way. So we built BigMemory and put it in our server. Now you can take our pure Java Terracotta Server, have it use that 32 GB ram and still get less than 1 second pauses. Practically No GC.

For anyone who spent countless hours like I have tuning GC is going to love this. We ran and messed around with every knob the Sun (now Oracle) gave use to tune GC. When we put in BigMemory, we just deleted all those settings. Our heap is small enough so that default Java settings are good enough!

End result? Our Terracotta Server with BigMemory enabled can achieve higher density will less servers and best of all you can get predictable latency out of them.

Check out BigMemory and let me know if it works just as well for you.

automagically cluster Lift web sessions with Terracotta

2010-02-17T07:53:00.000-08:00

Lift Web Framework Is a MVC Framework written in Scala, Which are two things I know exactly nothing about. Being that, I thought It would be a good example to show how easy it is to cluster web session using terracotta new express web session product. Here's a link to our Beta ...

Part 1: The Lift/Scala Thing...

So first I had to figure out what the deal with Lift is about. Good thing there's a fairly simple tutorial to create a basic HelloWorld lift webapp here. First let's download and install all the bits you need to write a Lift web app, fortunately for me it was just the Scala Eclipse Plugin.

Let us create a maven project:


mvn archetype:generate -U \
-DarchetypeGroupId=net.liftweb \
-DarchetypeArtifactId=lift-archetype-blank \
-DarchetypeVersion=1.0 \
-DremoteRepositories=http://scala-tools.org/repo-releases \
-DgroupId=com.examples.terracotta \
-DartifactId=clusteredCounter \
-Dversion=1.0-SNAPSHOT

cd clusteredCounter
mvn jetty:run

Now you got yourself a fully functioning hello world webapp. Let's add some code where we are actually saving and displaying session data.

add a Lift Snippet that would process url parameters add then to the session and then display all the session attributes:


package com.examples.terracotta.snippet

import javax.servlet.http._
import net.liftweb.http._
import net.liftweb.util._
import net.liftweb.util.Helpers._

import net.liftweb.http.SessionVar
import scala.collection.mutable.HashMap
import scala.xml._
import java.util.Enumeration
import scala.collection.mutable.HashSet

class ViewSubmission {


def showStuff : NodeSeq = {

var title = S.param("title").openOr("")
var url = S.param("url").openOr("")

S.servletSession.get.setAttribute(title, url)

var e = S.servletSession.get.getAttributeNames
val names = new HashSet[String]
while (e.hasMoreElements()) {
 val name = e.nextElement().asInstanceOf[String]
 names += name
 println(names)
}

<table> {
   for (name <- names) yield
     <tr>
      <td>Title</td>
      <td>{name}</td>
     </tr>
     <tr>
      <td>Url</td>
      <td>{S.servletSession.get.getAttribute(name)}</td>
     </tr>
}</table>

}

}

Edit the index.html to show edit and show session data:


<lift:surround with="default" at="content">
<table>
 <lift:snippet type="viewSubmission:showStuff" />
</table>
<form>
 <tr>
  <td>Title</td>
  <td>
   <input type="text" name="title" />
  </td>
 </tr>
 <tr>
  <td>Url</td>
  <td>
   <input type="text"  name="url"/>
  </td>
 </tr>
 <tr>
  <td>&nbsp;</td>
  <td><input type="submit" value="Add" /></td>
 </tr>
</form>
</lift:surround>

Run mvn -Djetty.port=9999 jetty:run and another instance with

mvn -Djetty.port=9999 jetty:run. got to http://localhost:8888 and http://localhost:9999. you can only see the data you added to each local web session.

Now the fun part...

Part 2: Clustering the web session

First we need to add our express web session jar to the pom.xml as a dependency:


<dependency>
<groupid>org.terracotta.session</groupid>
<artifactid>terracotta-session</artifactid>
<version>1.1.0-SNAPSHOT</version>
</dependency>

and also Terracotta Maven Repo information..

<repository>
<id>terracotta-snapshots</id>
<url>http://www.terracotta.org/download/reflector/maven2</url>
<releases>
  <enabled>true</enabled>
</releases>
<snapshots>
  <enabled>true</enabled>
</snapshots>
</repository>

Let us add our terracotta filter for clustering:


<filter>
 <filter-name>terracotta</filter-name>
 <display-name>Terracotta Session Filter</display-name>
 <!-- The filter class is specific to the application server. -->
 <filter-class>org.terracotta.session.TerracottaJetty61xSessionFilter</filter-class>
 <init-param>
  <param-name>tcConfigUrl</param-name>
  <!--
   <param-value> of type tcConfigUrl has a <param-value> containing the
   URL or filepath (for example, /lib/tc-config.xml) to tc-config.xml.
  -->
  <param-value>localhost:9510</param-value>
 </init-param>
</filter>

<filter-mapping>
 <filter-name>terracotta</filter-name>
 <url-pattern>/*</url-pattern>
</filter-mapping>

Now start our terracotta server, download here

Start are terracotta server and then run the jetty servers again. this time you can see session data on entered in on one server appear on the other.

Example Project

Performance Tuning Hibernate Second-level Cache

2009-06-28T00:34:00.000-07:00

Lately I've been spending ALOT of time performance testing and tuning of a hibernate app using a Terracotta implementation of Hibernate Second-level Cache. I thought it would be interesting to do a series of posts that walk through different use-cases and how we performance tuned them. While doing so, I thought it would also interesting to demonstrate various tools that can help with the tuning.

Before we continue, I think it is worth while to read my colleagues posts on performance testing here, here, and here. Their advice seems simple enough. But trust me, it's really easy to fall off the wagon when it comes to this stuff. The work itself is very tedious, so it's easy to convince yourself that "Yeah, this patch is it, this is the final run", or "I know the bottleneck is the CPU for sure! Why bother looking at the memory and/or disk characteristics." Assumptions and unfounded optimism can easily get you rat-holed to a dead end. Bottom line: don't assume anything and collect everything.

The project that we are going to look at is a simple hibernate app. The app itself just warms up the 2nd-level cache and then accesses the cache, and depending on the use case writes, in a non-partitioned fashion in the following modes: read-only, read-mostly and read-write.

the various tools I used to tune is:
- nmon a basic tool to look at cpu, memory, disk and network metrics.
- jmap,jhat,jstat to analyze heap and garbage collector metrics, these tools are included in the jdk.
- VisualVM, very cool tool to look at JMX Beans, nmon type characteristics as well as a light-weight thread monitoring tool.
- Terracotta Dev Console, great tool for looking at terracotta specific characteristics as well as general info (cpu, memory etc.)
- Good old fashion thread dumps (kill -3 java_pid)

Ok, so that's our test and tools. More of the nitty-gritty to come :)

Carrying over Eclipse Plugins from Ganymede to Galileo

2009-06-23T23:28:00.001-07:00

Looks like a new version of Eclipse is coming out!

Over the course of using Ganymede I accumulated various plugins that I want to carry over to Galileo. I figure there must be a way to export and then import those plugin Urls.

So first I launched my Ganymede and go to Help-> Software Updates..
Then click "Manage Sites.." You should see the following Dialog:

Click the checkboxes for which site Urls you want to export and Click "Export" and same the file. I saved the suggested bookmarks.xml to my Desktop.

Now I downloaded the Galileo RC4 (Galileo final release comes out tommorrow) and launched it.
Now go to Help->Install New Software..

Click on the link "Available Sofware Sites" like the one in the screenshot below:

Then select Import and select the file you saved from Ganymede. Now make sure you have "Hide items that are already installed" Otherwise it will take a long time to come up with the plugin list.

Now just install the plugins you want to install. Can't make any promises that it will actually work with Galileo :)

Terracotta Cluster Events: A simple example

2009-06-21T19:26:00.000-07:00

My friend instant messaging me this weekend about this app he is writing using Terracotta. He is concerned about when a node leaves/joins and terracotta cluster and was doing some really weird stuff with the JVM Shutdown hook. I decided to do a quick blog post about Terracotta Clustered Events which is a simple way for an app to listen for terracotta cluster wide events.

So first i'll generate a new project using the pojo-archetype:


mvn org.apache.maven.plugins:maven-archetype-plugin:2.0-alpha-5:generate \
-DarchetypeGroupId=org.terracotta.maven.archetypes \
-DarchetypeArtifactId=pojo-archetype \
-DarchetypeVersion=1.5.0-SNAPSHOT \
-DgroupId=org.terracotta.examples \
-DartifactId=clusteredapi-examples \
-Dversion=1.0.0 \
-DremoteRepositories=http://www.terracotta.org/download/reflector/maven2

The the following prompt appears:


[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Default Project
[INFO]    task-segment: [org.apache.maven.plugins:maven-archetype-plugin:2.0-alpha-5-parent-patch-SNAPSHOT:generate] (aggregator-style)
[INFO] ------------------------------------------------------------------------
[INFO] Preparing archetype:generate
[INFO] No goals needed for project - skipping
[INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.
[INFO] [archetype:generate]
[INFO] Generating project in Interactive mode
[INFO] Archetype repository missing. Using the one from [org.terracotta.maven.archetypes:pojo-archetype:1.5.0-SNAPSHOT] found in catalog local
[INFO] Using property: groupId = org.terracotta.examples
[INFO] Using property: artifactId = clusterapi-example
[INFO] Using property: version = 1.0.0
[INFO] Using property: package = org.terracotta.examples
Define value for property 'description': :

Now enter in a description, I typed in "This is an example project to demonstrate java clustered api" and then was prompted again and Typed in "Y"
Now we have a project to test out terracotta clustered api.

The clustered event api interfaces are available from the terracotta api jar, let's include that in our pom.xml


<dependency>
           <groupid> org.terracotta.api</groupid>
           <artifactid>api</artifactid>
           <version>1.0.0</version>
</dependency>

Change the process class to have a count of 4, so we run 4 nodes from our tests.


<plugin>
            ...
                <processes>
                    <process nodename="app" count="4" jvmargs="-Xmx20m">
                        <classname>org.terracotta.examples.App
                        </classname>
                    </process>
               </processes>
</plugin>

Now that we have our app I want it to be clustered aware. In this example, I'm not going to do anything interesting except print out when node's joins and leaves. So I added the following listener and implementation to my existing App class:


public class App implements DsoClusterListener {

   @InjectedDsoInstance
private DsoCluster cluster;

public void registerListener() {
    this.cluster.addClusterListener(this);

}

...
public void nodeJoined(DsoClusterEvent dsoclusterevent) {
    System.out.println("nodeJoined Event about node: " + dsoclusterevent.getNode());
}

public void nodeLeft(DsoClusterEvent dsoclusterevent) {
    System.out.println("nodeLeft Event about node: " + dsoclusterevent.getNode());
}

public void operationsDisabled(DsoClusterEvent dsoclusterevent) {
    //;
}

public void operationsEnabled(DsoClusterEvent dsoclusterevent) {
    //
}

And then call registerListener from the main() function:


App app = new App();
app.registerListener();
app.addMessage("Hello, world");

now run the app with the following command to see the node events


mvn clean install
mvn tc:run

Much much more simpler then registering ShutDown hooks. The project is here

Terracotta Integration Module with Maven

2009-06-20T11:29:00.000-07:00

TIMs or Terracotta Integration Module is a way to package classes and configuration for certain interfaces and products so that you can add terracotta clustering without much effort. For example, we have a TIM for ehcache so that anyone who uses ehcache can get terracotta clustering just by including the TIM http://forge.terracotta.org/releases/projects/tim-ehcache/

Sometimes using archetypes in maven is a bit unweilding, so first we are going to check out the correct versions and make sure everything is in place.

Let's check out the maven-archetype-plugin first:


svn co http://svn.apache.org/repos/asf/maven/archetype/tags/maven-archetype-2.0-alpha-4/ maven-archetype-2.0-alpha-4
cd maven-archetype-2.0-alpha-4
mvn clean install

Now we want to checkout and install the tim-archetype plugin, is is what builds the directory structure and configuration for our TIM project.


svn co http://svn.terracotta.org/svn/forge/projects/tim-archetype/trunk/ tim-archetype
cd tim-archetype
mvn clean install

To create a TIM project you need to run the following command:


mvn org.apache.maven.plugins:maven-archetype-plugin:2.0-alpha-4:generate \
  -DarchetypeGroupId=org.terracotta.maven.archetypes \
  -DarchetypeArtifactId=tim-archetype \
  -DarchetypeVersion=1.5.0-SNAPSHOT \
  -DremoteRepositories=http://www.terracotta.org/download/reflector/maven2 \
  -DgroupId=org.terracotta.modules.memcached \
  -DartifactId=tim-memcached \
  -Dversion=1.0.0

this will prompt you to run the following command:


[INFO] Using property: groupId = org.terracotta.modules.memcached
[INFO] Using property: artifactId = tim-memcached
[INFO] Using property: version = 1.0.0
Define value for property 'package':  org.terracotta.modules.memcached.tim-memcached: :

Hit enter or type in the package name you desire and hit enter.


Define value for property 'package':  org.terracotta.modules.memcached.tim-memcached: : 
Confirm properties configuration:
groupId: org.terracotta.modules.memcached
artifactId: tim-memcached
version: 1.0.0
package: org.terracotta.modules.memcached.tim-memcached
 Y: :

Type in 'Y' to finish creating a newly minted TIM.

All set to go!

Back to Basics

2008-10-31T01:18:00.000-07:00

When doing a little refactoring on terracotta server code, I was reminded on why the basics ALWAYS matters.

The server side representation of both ArrayList and LinkedList mapped to a class called ListManagedObjectState. The underlying data structure for this class was a ArrayList. We got around to mapping LinkedList to a LinkedListManagedObjectState where the underlying data structure for that class is a LinkedList.

I wrote a little test and the following where the results:


ArrayList.add for 50000 objects took 424 ms.
LinkedList.add for 50000 objects took 198 ms.
ArrayList.addFirst for 50000 objects took 906 ms.
LinkedList.addFirst for 50000 objects took 109 ms.
ArrayList.addAt for 50000 took objects 105 ms.
LinkedList.addAt for 50000 took objects 132 ms.
ArrayList.clear for 50000 took objects 127 ms.
LinkedList.clear for 50000 took objects 49 ms.
ArrayList.removeFirst for 50000 took objects 847 ms.
LinkedList.removeFirst for 50000 took objects 70 ms.
ArrayList.remove for 50000 took objects 865 ms.
LinkedList.remove for 50000 took objects 68 ms.

If your mutating a lot more then accessing, then LinkedList is the way to go.
Basics do matter!

DSO Garbage Collection: YoungGen

2008-10-30T23:30:00.001-07:00

There are certain apps when clustered using Terracotta that generates short-lived DSO objects and lots of them. Which results in a lot of DSO garbage!

Taking the existing DSO garbage collector and running it more often will collect these short-lived objects, but the MARK stage could take a long time. For example, you'll have a Terracotta server array that has DSO objects on the server's cache as well as DSO objects faulted to disk. When the DSO garbage collector runs, it will fault in objects from the disk during the MARK stage. This may take awhile to run.

The DSO objects that are short-lived are probably in the server's cache. This is where YoungGen comes in. It's the same algorithm, except for the set of objects we are considering as GC Candidates. Instead of considering all the objects in the system and removing objects that are unreachable. We only consider the objects on the server's cache and remove objects that we can definitely determine as garbage.

DSO objects that cannot be collected in YoungGen is the following:

1. Roots
2. DSO Objects that in the server's cache that is being references by DSO Objects faulted to disk. This is marked in a subset called RememberMe set
3. Objects that are in the cache, but never been flushed to disk (i.e. really new objects).

enable the following properties and see if YoungGen DSO Garbage Collection works for you:


l2.objectmanager.dgc.young.enabled = true
l2.objectmanager.dgc.young.frequencyInMillis = 180000

Terracotta DSO GC Algorithm in a nutshell

2008-10-29T17:35:00.000-07:00

There maybe a few of you out there (when I mean few, hopefully a few million :)) that maybe interested in how Terracotta collects its Distributed Objects that is no longer in use. Ok here goes...

Objects that should be collected by DSO Garbage Collection are objects which are no longer referenced by any client (meaning the object has already been GCed locally on every client that held it). So now that the clients have no more use for this object, its up to the server array to dispose of it (i.e. remove it from the server array cache or terraocotta's persistent storage).

When DGC kicks in..

1. It starts monitoring for new objects being created and keep a set of those. This is used later to rescue GC candidates.

2. It does its initial mark, which is to get all the ObjectIDs in the system. And also our roots in the system. We remove the roots and well as object reachable by roots form GC Candidates set. The objects that are left are objects that are no longer referenced by anyone

3. Remember when monitoring new object references was turned on? Now new referenced objects that appeared in our GC Candidate set can be removed from that set. This means there were new references to those objects while our first marking was going on, so they need to be rescued and not collected anymore. If after this rescue there are no GC candidates, then GC is complete.

4. If there are still GC Candidates left, then the system is paused. For us this means pausing our ObjectManager so that new lookup or create requests
goes pending (i.e no new references are created). We now do another rescue and produce our final GC Candidate set for this GC Cycle. Then new reference monitoring is turned off.

5. finally we have a have a set to delete and pass that list onto a different thread that will delete the GC Candidates from the ObjectManager in batches.

Since the delete is staged on a different thread (SEDA arch). It is possible for another GC Cycle to kick in before all the delete request complete, which can in turn create more objects to delete and life goes on...

Up next: Young GC Candidates....