OPJ Tutorial: Evolution

The easiest way to appreciate the evolution problem is literally to imagine that the OPJ Virtual Machine really is running continuously and then imagine that there is a need to change the behavior of the application while it continues to run.  Changing the behavior will require us to change the definitions of some classes and/or objects that are already loaded and running in the OPJ Virtual Machine. Unfortunately, unlike in some other languages, such as Smalltalk, it is not possible to change the definition of a loaded class using the standard facilities of the Java language, so the OPJ platform has added some new features to make this possible.

In general, changing the definition of a class, for example by adding a new field, will require corresponding changes to all of the reachable instances of that class.  The change may also require that clients of the changed classes be changed. At the very least the change must be checked against all possibly affected classes. It is very important that the all the necessary changes to all affected classes and instances happen simultaneously, conceptually at least, and that all the changes are performed atomically.  In other words we don't want a client to see some instances in the old form and some in the evolved form. The requirement of atomicity is quite challenging to achieve in an executing virtual machine and of similar difficulty to implementing persistent threads.  Therefore, in the current OPJ prototype, evolution only takes place offline on a suspended computation in a persistent store. This prevents the evolution of 7x24 applications, but is adequate for many applications that can tolerate a brief shutdown.

Evolving the Currency Converter Example

Changing a Method Body

Returning to our currency converter example, assume that we want to make a simple change to the main method, namely to alter the prompt "Input command: " to "Input a command: ". We make the change to the source file and recompile and run the application again:

javac Main.java opj -store /home/user/jane/ccalc.pjc opjtutorial.ccalc.Main Input command:

But wait a minute, the application didn't change! What went wrong?  The answer is simple. The recompilation only affected the Main.class file in the file system of the development machine. It had no affect on the suspended computation in the persistent store, which still has the original class loaded. Since we can't unload that class, we have to evolve it instead. As a convenience OPJ provides a special version of javac, called opjc,  that compiles and evolves the class in one simple operation:

opjc -store /home/user/jane/ccalc.pjc Main.java opj -store /home/user/jane/ccalc.pjc opjtutorial.ccalc.Main Input a command:

Observe that the behavior has now changed to reflect the changes in the source file.

An important feature of this form of evolution is that the actual instance of the opjtutorial.Main class was updated with the new behavior; in particular, a new instance was not created. Therefore the identity of the Main.class object itself remains unchanged. This is important since it is (the identity of) this object that is recorded in the persistent root set.  The preservation of object identity is a general property of OPJ, and applies across checkpoints and across evolutions (and all other store management operations).

The opjc compiler is compatible with the javac compiler and behaves identically with respect to the rules for finding classes that are needed during compilation. This is, all needed classfiles are expected to be found in the file system via the classpath.  No attempt is made to locate classfiles in the persistent store if they cannot be found in the file system.

Adding New Functionality

As before, we can try to compile and evolve the class in one operation using opjc:

opjc -store /home/user/ccalc.pjc Main.java Instance data fields added in the substitute class opjtutorial.ccalc.Main No conversion class available for class opjtutorial.ccalc.Main There are instances of this class in the store Do you want to rely on default conversion (d) or cancel (c)? Please enter d or c :

Note that the evolution tool observes that we have added the field history to the class and warns us that the instance of this class in the persistent store does not contain this field and requires conversion.  The simplest way to convert a class is called default conversion.  What default conversion does is preserve the values of fields that have the same name and type, and assign new fields the default value specified by the Java Language Specification. Since history is a reference type, it will receive a default value of null.  Since we defined history to be initialized to new LinkedList(), default conversion won't work in this case. If you go ahead with default conversion and run the application again you will get a null pointer exception from the commands that access the history list. You might wonder why default conversion doesn't apply the programmer defined initialization code,. Perhaps one day it will, but for now you have to fall back on the general mechanism of a programmer-defined conversion method.

The general idea behind conversion methods is that they convert an object instance from the old version of the class definition to the new one.  This is often a simple transformation, as in our current example, but can become quite complex and involve restructuring of the class hierarchy or replacing extensive data structures, e.g. a list with a tree. The evolution system can handle arbitrary transformations involving many classes, but the amount of planning and programming involved is proportional to the complexity of the change.  Conversion methods are written in the Java language, which introduces an immediate problem because a conversion method usually needs to refer to both the old and the new class definitions at the same time. This isn't possible in standard Java and is another reason that OPJ provides its own variant of the Java compiler, opjc. To minimize the impact on the language definition, the old version of a class is denoted by a special suffix, $$_old_ver_, to the name of the class.  This is a perfectly legal identifier in the Java language, but extremely unlikely to be chosen in normal programming.

Conversion methods come in several forms depending on the conversion that is required. The simplest form, which we will be using is called convertInstance, and has the following signature:

public static void convertInstance(Type$$_old_ver_ oldInstance, Type newInstance);

This method converts objects of type Type and is applied automatically by the evolution system to all instances of class Type that are found in the persistent store. In addition, with this variant, default conversion is applied automatically first, which means that the conversion method need only deal with the fields that actually need manual conversion.  To get back to our example, all we need to do in the conversion method is initialize the history field of newInstance to an empty list:

public static void convertInstance(Main$$_old_ver_ oldInstance, Main newInstance) {   newInstance.history = new LinkedList(); }

Where should the definition of a conversion class go? The obvious and simple choice is to put it in the Main class itself.  In fact, if we do not put it there we will be unable to compile the code, because the history field is marked private. In general, modulo access issues, you can put conversion methods in any class.  You must use opjc to compile conversion classes because of the need to access both version of the class definition. You must also tell opjc the name of the conversion class with the convclass option.

opjc -store /home/user/jane/ccalc.pjc Main.java -convclass opjtutorial.ccalc.Main

Now if we run the application we can access the new functionality:

opj -store /home/user/jane/ccalc.pjc opjtutorial.ccalc.Main Input command: h Exchange rate history: 1.65 Input a command: x1.7 Exchange rate set to 1.7 Input a command: h Exchange rate history: 1.7 1.65 Input a command: q

Another simple change that we can make is to eliminate the exchangeRate variable by merging it with the history list as the first element.  To achieve this requires access to both the old and the new instances and class definitions.  Ideally it would be possible to write the following conversion code:

public static void convertInstance(Main$$_old_ver_ oldInstance, Main newInstance) {   newInstance.history.add(0, oldInstance.exchangeRate); }

Unfortunately, this will not compile because the evolution system considers Main$$_old_ver_ as a separate class  and so access is denied to the oldInstance.exchangeRate field. This restriction may be lifted in a future release, but for the meantime the workaround is to use reflection as follows:

public static void convertInstance(Main$$_old_ver_ oldInstance, Main newInstance) {     try {         newC.history.add(0, oldC.getClass().getDeclaredField("exchangeRate").get(oldC));     } catch (Exception ex) {         throw new RuntimeException("unexpected failure to find exchangeRate field");     } }

This works because the evolution system exploits the ability to suppress access checks during reflective operations, thus allowing access to private fields.

Exceptions in Conversion Methods

More Complex Changes

More details on the facilities of the evolution system are provided in the documentation on the opjsubst tool, which is the actual evolution engine. In the above examples, opjsubst was invoked implicitly by the opjc compiler.