2.2.1   The Monterey Service Developer Pack

When installed via the update site into an existing Eclipse installation, Eclipse will first prompt for a restart. After that, you are ready to go!

When using the stand-alone Monterey Studio for the first time, you must configure a workspace path. The workspace is the directory where you want to keep all of your project files. This directory will be created for you if it doesn't already exist. Remember to select the 'use this as the default workspace' option, or this question will be asked every time you run Monterey Studio. On first launch the default view is the welcome screen - this provides a few helpful links to get started, including displaying this documentation in an Eclipse pane, a link to start the tutorials, or getting rid of the welcome screen to begin with a blank workspace. The welcome page can re-opened at any point by clicking Help -> Welcome, while this user-guide and other help, tutorials and documentation can be accessed stand-alone by Help -> Help Contents or inside an Eclipse pane with Help -> Dynamic Help.

If you have installed Monterey into an existing Eclipse or SpringSource Tool Suite, the welcome screen is not available, but you can access the Monterey help via the Help -. Help Contents or Help -> Dynamic Help menu options as normal.

2.3.1   Monterey Service Development

The simplest way to get started is to use the new project wizard, which can be found by navigating to File -> New -> Monterey Service Project. Write the service interface and service class implementation with your desired business logic, then you are ready to run...

For further details on the API used throughout the templates, see Developing Monterey Applications.

2.3.2   Monterey Classic Development

To write a Monterey classic application using the messaging-based API, use the wizard at File -> New -> Monterey Classic Project. The stubs template provides a simple skeleton project for writing the client-gateway and segment-service code.

Other templates are based around some of the tutorial projects, generating fully-formed projects which can be used as examples or as a starting point for creating your own applications.

For further details on the API used throughout the templates, see the section on Developing Monterey Applications.

2.4.1   Running Your Application

To run a Monterey Application, you must first create a Monterey Application launch configuration. This may be done in the Run Configurations window, which can be accessed by clicking Run -> Run Configurations....

Separate run configuration types are provided for simulating a Monterey network on the local machine versus deploying the application onto a live network. You can create a new configuration by double-clicking Monterey Simulation or Monterey Deployment on the left-hand pane or, to re-use another projects configuration as a start point for a new configuration, right-click another configuration and select Duplicate. You may wish to rename run configurations if you're managing more than one project and have many of them.

The primary field for a run configuration is the Monterey Application (found on the first tab of the run configuration dialog that should be displayed initially), which specfies the application deployment descriptor file to be used. This defines such things as which OSGi bundles are required, which Java classes comprise the Monterey gateway and service components, what segments should initiallly be created, and so on.

For Monterey Deployment configurations, it is also necessary to specify how to connect to the running network's control plane. In many situations this requires choosing Remote Monterey Server over Web API from the drop-down list, and providing the necessary URL and user credentials provided by the network administrator.

The run configuration can be run straight from this window. There is also the Run menu, which offers a choice from a selection of recently used configurations under Run History or to go to the Run Configurations window to select and run a configuration from there.

2.4.2   Controlling Monterey

Once the application is running, the easiest way to set-up the network is using the Quick Setup option found on the drop-down menu for the Resources view. These options allow the rapid provision, roll-out and configuration of a network of many machines to quickly start testing an application or act as a starting point for a custom network.

• The Resources view shows the nodes in the network, arranged hierarchically based on their location; can be used to provision and roll-out new nodes, and to make dynamic changes such as manual offloading or rerouting.
• The Nodes view shows a list of all nodes in the network with information about each (e.g. workrate, topology, etc).
• The Segments view gives a variety of details on the various segments and their performance. It also provides access to manual migration and the colouration of segments throughout the application to make segments easier to trace.
• The Bots view is used for controlling bots, if they exist in your application, including their segment and group weighting, their overall activity and scripting.
• The Policies view is used for configuring automatic migration policies that keep the network performing optimally.
• The Network Topology view displays the structure of the Monterey overlay network graphically.
• The Geographies view showing the capacity and activity at each of the configured geographic regions. Inter-geography migrations are shown in real-time on this view.

Further details on the management console, manual deployment (i.e. without the management console) and other run-time considerations can be found in the section on Running Monterey Applications.

3.1.1   Installing Monterey Studio

Monterey Studio can be used as both a standalone product and as Eclipse plug-in features for an existing Eclipse installation. The standalone product is recommended. These installation instructions are for the standalone product.

Your product distribution should include a ZIP archive for your platform named similar to the following:

Monterey-[version]-[platform].zip

Unpack this archive to a suitable install location on your system.

The contents of the install package are as follows:

• docs/
  • getting-started.txt
  • release-notes.txt
  • licence.txt
  • Monterey-Users-Guide.pdf
  • Monterey-javadoc/
    • Standalone HTML documentation for the Java API classes for offline browsing.
• MontereyStudio/
  • Contains the IDE executable; dive right in if you want to start developing, or browse the tutorials or online user guide.
• MontereyManagementConsole/
  • The standalone management console for connecting to and managing a monterey deployment.

To get started, locate and run the MontereyStudio executable in the MontereyStudio folder. The first time you launch Monterey Studio, you will be prompted for a location where your workspace data will be stored. We recommend you accept the default workspace location, which should be a directory named monterey-workspace in your home directory.

3.1.2   Installing Monterey into Eclipse

Monterey can be installed using Eclipse's "Install New Software" functions. This also applies to Eclipse derivatives such as SpringSource Tool Suite. Eclipse 3.6.1 and SpringSource Tool Suite 2.6.0 are supported by Cloudsoft[*].

Before installing Monterey, you will need to install the Groovy-Eclipse plugin, by following the instructions at http://groovy.codehaus.org/Install+Groovy-Eclipse+Plugin

Select the Help -> Install New Software... menu option. Press the Add button in the top-right of the Available Software dialog box, and enter the following information into the Add Repository dialog:

The Monterey update site is at:

• For SpringSource Tool Suite 2.6: http://developers.cloudsoftcorp.com/download/eclipse/update-sts26
• For Eclipse 3.6: http://developers.cloudsoftcorp.com/download/eclipse/update-e36

Click OK. Eclipse will now display the software packages available from this location:

• Monterey Service Developer pack: offers the full power of writing Monterey applications, using either the service development interface-based approach or the classic messaging-based API. This should be installed by developers writing services to run on the Monterey platform.
• Monterey Client Developer pack: supports Monterey Client Development, for connecting to and using a Monterey service from within another application. This should be installed by developers who wish only to consume existing services.

Having chosen the features to install, click Next and follow the instructions on-screen. Eclipse will prompt you to restart; once this is done, Monterey is installed, and its feature will be available to use in Eclipse.

3.3.1   Introduction

Monterey allows a service developer to focus on their core business logic, by writing an interface that is exposed to clients of the service. The underlying messaging APIs are entirely hidden from the service and client developers.

In Spring terminology, each segment is a bean that encodes the application's business logic. Each bean exposes an interface through which it can be accessed by client-code. Importantly, these beans are mobile: individual beans can be suspended, migrated to a different node, and resumed without impacting the rest of the application. In this distributed application, calls from the client are automatially serialized and routed to the appropriate bean where they are processed, and the response sent back to the client (a useful analogy for this is RMI remoting). All the client and service developers need do is ensure that the parameters and return types of their methods are serializable.

This approach is best illustrated with an example. Consider a Hotel Booking Availability service for Foo Hotels, a world-wide hotel chain. The service can be logically partitioned with each hotel being a segment. Each segment is stateful, and is the definitive source of hotel availability information. Each segment exposes an interface for booking a room and cancelling bookings in that particular hotel. The business logic for each hotel is responsible for tracking the room bookings, and accepting/rejecting booking requests. A client consuming this service first looks up the particular hotel and then makes calls on the hotel interface to make its desired bookings.

3.3.2   Creating a Monterey service

To create a Monterey service, the best starting point is the Monterey Service Project wizard. This will normally create a pair of projects, one containing the interface, and the other containing the implementation. This separation allows the light-weight interface project to be bundled and given to the client-side development team (who do not need access to the implementation.) You can however override this behaviour and place all the classes into the same project.

The Monterey Service Project is available under the File -> New menu in the Java perspective. Alternatively select the File -> New -> Project... menu option to see a list of all project types; expand the Monterey Middleware group, select Monterey Service Project, and then click Next. You will then see the Create a Monterey Service Project dialog:

The wizard asks for the service interface and implementation to be used for each segment. The wizard then creates the required classes and meta-data files to allow this application to be deployed to a Monterey network.

You have the following options on this dialog:

• Project name: the name of the new project. If the option is selected to "create api and implementation in separate projects" then two projects will be created with this project name prefix, with suffixes ".api" and ".impl".

• Java package root: the package name of the new classes and interfaces. This is normally the same as the project name but can be changed if needed. the public interfaces and classes are created in this package, while the service implementation classes are generated in a sub-package with suffix ".impl".

• Monterey classes prefix: an optional prefix to be applied to the names of the supporting classes (it does not apply to the service interface nor the service class names.)

• Service interface: the name of a Java interface, which will be the externally-facing API of the Monterey network service. If "create service interface" is selected then this name must be unqualified and the interface is generated in the "java package root"; otherwise the interface name should be fully qualified.

• Service class: the name of a class that will implement the interface, and thus contain the segment's business logic. This class's code will be executed inside the Monterey network. If "create service class" is selected then this name must be unqualified and the class is generated in the ".impl" sub package; otherwise the class name should be fully qualified.

• Create service interface (uncheck to use pre-existing): check this box to create an empty interface with the above name; uncheck if you do not require this interface to be created (if, for example, you have a pre-written interface.)

• Create service class (uncheck to use pre-existing): check this box to create an empty implementation class with the above name; uncheck if you do not require this class to be created (if, for example, you have a pre-written interface.)

• Create api and implementation in separate projects (with suffixes .api and .impl): check this box to create two projects - one project with the suffix .api containing the interface, and one with the suffix .impl containing the implementation class. Unchecked, a single project is created and all types are placed in this project.

• Support external proxy LPP connections: check this box to create an LPP-client-gateway that supports proxy LPP connections. This is necessary for external clients that are behind a firewall or that have a private IP. They require a proxy connection via an LPP in the monterey network. See the section Client-side firewalls and private IPs.

• Create and register a bot: check this box to create an empty bot class. This also creates an

  LPP-client-gateway that registers the bot. See the section Testing and Bots.

• Initial segments: this is the list of initial segments that is placed in the application descriptor configuration file, and will be started when the service is deployed into a Monterey network. For more information on segments, refer to Essentials of a Monterey Application.

When you click Finish, the project(s) and types will be created.

The next step for the developer is to flesh out the service interface and implementation with their business logic. Care should be taken to ensure that all parameters and return values are serializable.

That's it. The service developer can now deploy and manage the Monterey application. The auto-generated deployment descriptor in the conf directory is used when deploying the application.

3.3.3   Providing your service to client-side developers

To allow client-side developers to use the new Monterey service, the API code needs to be made available. Simply generate a jar for the .api project and make this available to client-side developers.

You will need to supply the API to client-side developers for them to be able to take advantage of your service on a Monterey network. Simply generate an OSGi bundle (i.e. a Jar) for the api project. The standard way to do this in Eclipse involves:

1. Right-click on the api project and select Export...
2. Expand the Plug-in Development group and select Deployable plug-ins and fragments. Click Next.
3. Alter any of the export options if required - you must at least choose a destination to export to.
4. Click Finish.

The client-side developers can then take the exported JAR bundle, and import it into their project. It can then be used as described in Client-side developers.

You may find it convenient during development to provide a bundle of the impl project also. This bundle will contain the LocalServiceLocatorImpl class - an implementation of the service locator that keeps the services in-process. This will allow the client-side developers to test against the service without requiring a Monterey network or simulator.

3.3.4   Client-side developers

A client-side developer can connect to a Monterey network and call the Monterey Service. The client uses a service locator to lookup the desired segment; the returned object exposes the service interface, and transparently routes the method calls over the Monterey network to call the implementation class.

You can start your Monterey client project for any kind of Eclipse Java project. You only need to add the Monterey nature to this project. To do this, right-click on the project in the Eclipse Project Explorer view, and select Configure -> Convert to Monterey Project.

The client-side code needs two jars: first, a Monterey jar, and second, the Monterey service APIs. It is the responsibility of the service developer to publish the service API Jar. The Monterey jar can be obtained by right-clicking on the project and selecting Monterey Middleware -> Add Monterey Jar (if this option is missing, see above to find out how to make your project a Monterey project.) This will copy the monterey-embedded-node.jar file into the root of your project. You should ensure this is added to your build path, moving it if necessary.

Connecting to a Monterey service involves instantiating a MontereyNetworkEndpoint and a ServiceLocatorImpl. The MontereyNeworkEndpoint supports authentication and message-passing; the ServiceLocatorImpl allows one to lookup a desired segment to get a proxy for the remote service instance. The client-code looks up the desired segment, and invokes methods on that segment. Method calls are synchronous: they block until the server-side has executed the method and returned the result, or until any exception thrown by the service is returned to the client.

The credentials used to connect to the Monterey network must match a user in either the client or admin role (as configured in the web-users.conf).

Below is some example Java code for a client using a hotel booking availability service:

import com.cloudsoftcorp.monterey.servicebean.access.api.MontereyNetworkEndpointImpl;
import com.cloudsoftcorp.sample.booking.BookingAvailabilityServiceLocatorImpl;
import com.cloudsoftcorp.sample.booking.BookingAvailabilityServiceLocator;
import com.cloudsoftcorp.sample.booking.BookingAvailabilityService;

public static void main(String[] args) {
   ...
    MontereyNetworkEndpointImpl endpoint = new MontereyNetworkEndpointImpl();
    endpoint.setManagementNodeUrl("http://thehostname:8080");
    endpoint.setUsername("guest");
    endpoint.setPassword("password");
    endpoint.setLocation("GB-LND");
    endpoint.start();

    BookingAvailabilityServiceLocator serviceLocator = new BookingAvailabilityServiceLocatorImpl(endpoint);
    serviceLocator.getHotel("The Ritz").createProvisionalBooking(checkInDate, checkOutDate);
    ...
    endpoint.shutdown();
}

An example of the Spring configuration for instantiating the service locator is shown below:

<bean name="montereyEndpoint" class="com.cloudsoftcorp.monterey.servicebean.access.api.MontereyNetworkEndpointImpl" init-method="start" destroy-method="shutdown">
    <property name="managementNodeUrl" value="http://thehostname:8080"/>
    <property name="username" value="guest"/>
    <property name="password" value="password"/>
    <property name="location" value="GB-LND"/>
</bean>

<bean name="bookingAvailabilityLocator" class="com.cloudsoftcorp.sample.booking.BookingAvailabilityServiceLocatorImpl">
    <constructor-arg ref="montereyEndpoint"/>
</bean>

Note that calls to serviceLocator.getService(String) are cheap - they do not involve any remote communication.

3.3.5   Mobility

Services are designed to be mobile - that is, they can be moved around the Monterey network, relocated to different nodes in the network. This may happen by a manual operation by a network administrator, or in response to decisions made by policies (for example, provisioning extra nodes and moving processing onto new nodes in response to increased demand.)

The service developer gets mobility for free if their class is serializable. When migrating a segment, the default behaviour is to serialize the implementation instance, send it across the wire to the new node, and deserialize it.

In many cases, standard serialization is sufficient. However, in some cases it is necessary to know when migration is occurring - you can listen for lifecycle events that happen before and after a service is moved. For example, one may have to open/close connections to external resources or start/stop additional threads.

To receive these lifecycle events, your service implementation class can use the @PreSuspend and @PostResume annotations provided by com.cloudsoftcorp.monterey.servicebean.annotation. These annotations can be applied to void methods with no arguments. The @PreSuspend annotated method is called by Monterey immediately before the service instance is serialized. The @PostResume annotated method is called by Monterey after the service instance is deserialized at its new location, and before Monterey resumes calling the service's methods.

If multiple methods are annotated with @PreSuspend or @PostResume, they will all be called but the order is not guaranteed. If a method is annotated that requires parameters then it will not be called.

Alternatively, your service implementation class can implement the interface SuspendableBean. This interface provides two methods:

• void suspend()
• void resume()

The suspend method is equivalent to using the @PreSuspend annotation; the resume method is equivalent to using the @PostResume annotation.

If annotations and the SuspendableBean interface are used simultaneously, then the annotated methods will be called first and then the interface's suspend or resume methods. The suspend() and resume() methods are guaranteed to be called exactly once, even if they are also annotated.

The typical lifecycle of a segment bean is: * On segment initialization at node X:

• instantiate the bean.

• When there is a need to move a segment (hosting your service) to another node Y in the network:
  • stop calling methods on the instance; if a method call is in progress, Monterey will wait for this method call to complete.
  • call all @PreSuspend annotated methods on the instance at node X
  • call suspend() on the instance at node X
  • serialize the bean to a binary data stream and discard the original object
  • send the data stream to node Y
  • deserialize the bean on node Y
  • call all @PostResume annotated methods on the new instance at node Y
  • call resume() on the new instance at node Y

3.3.6   Customizations

The auto-generated SegmentService class extends AbstractBeanSegmentService, which provides the default behaviour described above. The SegmentService class can be changed to customize this behaviour. The three most important methods for overriding in SegmentService are:

• newBean: called exactly once, when the segment is first created, to create the segment implementation.
• resumeBean: called after each migration, when resuming the segment-service.
• suspendBean: called before each migration, when suspending the segment-service.

A powerful use-case is to use an existing class without modification. This is demonstrated in the code snippets below for a pre-existing class FooImpl:

@Override
protected Foo newBean(String segment) {
    return new FooImpl();
}

@Override
protected Serializable suspendBean(Foo bean) {
    return bean.getMemento();
}

@Override
protected BookingService resumeBean(Object suspendedState) {
    return new FooImpl((FooMemento)suspendedState);
}

3.3.7   Auto-generated code

The files generated by the wizard are listed below. In common use-cases, most of these files can be safely ignored by the developer:

• The service interface and service class.
• The ServiceLocator interface: this is used by clients to lookup the desired segment. Note that the getService(String name) method can be renamed as desired (e.g. to getHotel).
• The ServiceLocatorImpl: this class implements the ServiceLocator interface, to be instantiated by clients.
• The deployment descriptor: this file, generated in the conf directory, describes the application. It is used when deploying the application.
• The SegmentService: this class acts as a bridge between the Monterey network and your service, mapping between the underlying messaging-based API to the user-defined service implementation.
• The Gateway: this class is the code to be deployed at LPPs within the Monterey network. It is generated only if either bots or proxying LPPs are enabled.
• LocalServiceLocatorImpl: this is an alternative implementation of the service locator interface; it is a convenience to aid unit testing of the application. All segments are instantiated and executed locally (i.e. in the current process) rather than being run in a distributed Monterey network.
• META_INF/OSGI_INF/serviceFactory.xml: this is an OSGi declarative service. It register an instance of SegmentService.Factory, which is used to instantiate an instance of the SegmentService for each segment.
• META_INF/OSGI_INF/gatewayFactory.xml: this is an OSGi declarative service. It register an instance of Gateway.Factory, which is used to instantiate an instance of the gateway for each LPP within the Monterey network. It is generated only if either bots or proxying LPPs are enabled.
• META_INF/MANIFEST.MF: this is the OSGi bundle manifest. The project(s) generated by the wizard are Eclipse plugin projects, and are thus OSGi bundles.

3.3.8   Client-side firewalls and private IPs

When a client-side developer connects to a Monterey network using the MontereyNetworkEndpoint, by default a server socket will be opened at the client-side. This allows nodes in the Monterey network to open socket connections to the client-side endpoint for efficient direct communication. However, if the client-side endpoint is on a machine with only a private IP or is behind a firewall, this approach is not possible.

The client-side developer can configure the endpoint so that it opens a single socket connection to an LPP in the monterey network. This is done by setting the additional property hasPrivateIp:

MontereyNetworkEndpointImpl endpoint = new MontereyNetworkEndpointImpl();
endpoint.setHasPrivateIp(true);

On the server-side, one or more LPP must be configured to accept these connections.

There needs to be a suitable LppClientGateway class deployed with the monterey service, and LPP(s) need to be rolled out. This can be done automatically in the Monterey Service Project wizard by checking the box Support external proxy LPP connections.

This auto-generates several artifacts in the project. First, there is an LppClientGateway which includes the following code:

public class MyExampleGateway extends BeanLppClientGateway {
    public static class Factory implements LppClientGatewayFactory {
        public LppClientGateway newClientGateway() {
            return new MyExampleGateway();
        }
        public LppStateBackup newClientGatewayBackup() {
            return null; // Unsupported
        }
    }
    public MyExampleGateway() {
        setSupportsProxyingLpp(true);
    }
}

There is an OSGi declarative service file auto-generated in the project's META-INF/OSGI-INF directory. Lastly, the deployment descriptor configuration file identifies that this LPP implementation should be used.

3.3.9   Future enhancements

This area is one of the main focuses for future versions of Monterey. We want to make it as easy to use and powerful as possible, and make the integration with Spring smoother. There are a number of features under consideration; the opinions of the community would be most welcome:

• Integration with Spring Bean Factories for instantiating and configuring each segment.
• Simpler support for exposing different interfaces at different segments.
• Support for asynchronous calls to the service.
• Support for the service to instigate communication with a client, and to broadcast to all clients (e.g. a listener pattern for clients).
• For the suspend/resume methods, support using the Spring configuration file to identify the methods.

3.4.1   Introduction

Developers can write Monterey Applications using the classic message-based APIs. Writing an application comprises the following activities:

1. Decide how the application state should be decomposed into independent segments.
2. Implement the Mediation Segment Service class that will run on the network's Mediator nodes, and encapsulates the business logic to handle mediation requests for an individual segment.
3. Implement the LPP Client Gateway class that will run on the netork's LPP nodes and provides the means for external clients to interact with the distributed application.
4. Bundle the Monterey classes (along with any additional code, such as views, help, etc.) into one or more OSGi bundles, using OSGi declarative services to advertize the factories for the segment service and client gateway to the runtime fabric.
5. Write an application deployment descriptor configuration file which specifies the segment service and client gateway factories to use, along with desired resilience properties, segments, and other configuration information.

The New Monterey Classic Project Wizard, described below, can automate much of the above work, but they are described here for completeness. The Monterey network also introduces additional nodes, between the LPP and Mediator nodes, in order to take care of routing messages between the services (2) and gateways (3), preserving consistency while segment migration and network topology changes are being made. These nodes are described elsewhere; although crucial to the operation of the Monterey network, they are mostly invisible to the application developer.

3.4.2   The Mediation Segment Service

Each segment is allocated a single associated MediationSegmentService instance, residing on a given Mediator node in the Monterey network. A single Mediator will host many such services, each configured to process mediation requests for a different segment. Obviously a MediationSegmentService will only see messages for the segment is it processing.

A MediationSegmentService is a process that lives on the network, that can receive messages from clients, process them, and if necessary reply to specific or all clients about the given segment on the network.

The MediationSegmentService interface (in com.cloudsoftcorp.monterey.network.api) looks like:

public interface MediationSegmentService {
    void doMediation(SenderReference sender, Object data);
    void initialize(SegmentServiceContext context, Object stateToResume);
    Serializable shutdown();
}

We can clearly see the responsibilities of a MediationSegmentService. The most important method is doMediation. This method is called when the Mediator receives a message in the corresponding segment.

When the segment service is initialized it is passed the existing state (if any) and a SegmentServiceContext object to allow it to communicate; and it provides the following methods:

public interface SegmentServiceContext {
    public void send(SenderReference ref, Serializable data);
    public void broadcast(Serializable data);
    public void error(String message, Throwable optionalException);
    public String getSegment();
}

The context instance is passed to the service's initialize method, which also provides a convenient hook for the service class to conduct any bootstrapping activities.

The Monterey Middleware framework guarantees that no messages will be delivered to the service (doMediation will not be called) until after the initialize method has completed.

The initialize method is one of the lifecycle methods that together permit the transfer of state between pre- and post-migration instances of the segment service:

• The shutdown method can optionally return the service's current state in a serializable memento that can be sent across the network to the new Mediator.
• At the new Mediator node, the memento is deserialized and passed to the new segment service's initialize method via the stateToResume parameter.

Note that the initalize method method takes an Object, but shutdown must return an instance of Serializable. This asymmetry is intentional, and reflects the minimum requirements for each method.

3.4.3   The LPP Client Gateway

At the client facing end of the network are the LPP nodes, providing a gateway onto the Monterey network itself. In the chat example from the tutorial, the gateway provided a web interface to end users so they could chat.

More technically, these nodes run code that conforms to the LppClientGateway interface (in com.cloudsoftcorp.monterey.network.api):

public interface LppClientGateway {
    void onPrivateMessage(String userRef, String segment, Object data);
    void onPublicMessage(String segment, Object data);
    void initialize(ClientGatewayContext context, Object stateToResume);
    Serializable shutdown();
}

The methods are fairly self explanatory. initalize is called on creation, shutdown on termination, onPublicMessage and onPrivateMessage are called when a message arrives at the LPP node, dependant on the type of message (i.e. public messages are the broadcasts).

As above in the mediator case, the LppClientGateway is provided with a context object to allow it to send messages on the network, specifically something that complies with the ClientGatewayContext interface (in com.cloudsoftcorp.monterey.network.api):

public interface ClientGatewayContext {
    public void sendMediationRequest(String userRef, String segment, Serializable data);
    public void error(String message, Throwable optionalException);
    public void sendToMonitor(Object data);
}

The sendMediationRequest method is used to communicate with the Monterey Network, and error is used to raise errors, forwarded onto the management console. sendToMonitor is used for out-of-band control messages, sent straight to the control plane, and this is used internally to announce bots for example.

If the gateway class implements the optional ManagementExtension interface, can optionally be implemented to indicate that gateway is capable of processing management messages, such as bot configuration commands:

public interface ManagementExtension {
    void onManagementMessage(Object data);
}

There is an example of handling ManagementExtentions in the bot section of the tutorial. There are two existing types of management messages - bot configuration messages, and the current (simulated) time-of-day as set in the Geographies View.

Finally, the gateway class can be run at one of two locations, either at a Vanilla or a Satellite LPP. The Satellite LPP is so client specific code can be run outside of the normal Monterey network. Client code (LppCLientGateway implementations) are unaware of precisely where they are running.

3.4.4   The New Monterey Classic Project Wizard

The New Monterey Classic Project wizard is capable of generating a "minimal" implementation of an entire Monterey application, including:

• non-functional (but valid) Java implementations of the required Monterey interfaces
• a bot class to simulate user activity for testing or demonstration purposes
• an embedded web-server for web-access to the gateways
• a deployment descriptor file to enable the application to be deployed onto a Monterey network through the Monterey Management Console

The wizard is often the most convenient starting point when developing a new Monterey application. It can be accessed by selecting New -> Project... -> Monterey Classic Project in the File menu or right-click menu in the Project View.

The most common use-case is to generate a project from the "Stubs" template. This creates the required project and directory structure, and creates empty (commented) implementations of the essential MediationSegmentService and LppClientGateway interfaces (described above). It also generates the OSGi meta-data MANIFEST.MF and OSGI-INF, and a conf directory containing a basic application deployment descriptor configuration file.

Other files that can be optionally generated include a very simple web-server and an empty bot implementation.

The comma-separated list of initial segments can be specified in the wizard. This can later be edited in the application deployment descriptor file.

An alternative code templates for generated a project is "Nothing". This creates the required Eclipse project and directory structure, but does not genreate any code or OSGi meta-data files.

3.5.1   Resilience with Monterey Service Development

To use checkpointing in the Monterey Service Development mode (as oposed to the classic mode), the developer needs to inject the SegmentServiceContext into the service class implementation. Some sample code for this is shown below.

Add the following code to the service class:

private transient SegmentServiceContext context;

public void setSegmentServiceContext(SegmentServiceContext context) {
    this.context = context;
}

Then in the service class, call context.checkpoint(this) wherever required. If using deltas, a SegmentStateBackup class is also required (see next section), whose getState() method must return an instance of the service class.

In the implementation package, you'll find an auto-generated SegmentService class, which is responsible for instantiating, suspending and resuming the segment instance. This can be modified to call the setter method when the instance is instantiated and when it has been deserialized. For example:

protected com.example.MyServ newBean(String segment) {
    MyServImpl result = new MyServImpl();
    result.setSegmentServiceContext(getContext());
    return result;
}

@Override
protected void notifyOfResume(com.example.MyServ val) {
    ((MyServImpl)val).setSegmentServiceContext(getContext());
    super.notifyOfResume(val);
}

3.5.2   State Backup

For the lossless resilience mode or when using checkpointing with deltas, the application developer must implement the SegmentStateBackup and LppStateBackup interfaces. Their purpose is to maintain a copy of the current state of a segment or LPP. For checkpointing using just the checkpoint() method, the latest checkpointed state is stored automatically without having to implement these interfaces.

The backup classes to use for the application are defined (and instantiated) by the MediationSegmentServiceFactory.newSegmentBackup and the LppClientGatewayFactory.newClientGatewayBackup methods. The factories are auto-generated when using the New Monterey Classic Project Wizard's "stubs" template, and the factory is registered as an OSGi service in the OSGI-INF meta-data. However, the methods newSegmentBackup and newClientGatewayBackup must be implemented by the user.

The default implementation is:

return null; // Unsupported

This needs to be changed to something like:

return new NameOfMySegmentStateBackup();

The lifecycle for a SegmentStateBackup or a LppStateBackup instance is that initialize() will first be called, supplying the current state. When that backup instance is no longer required, shutdown() will be called.

When the master segment-sevice calls checkpoint(mystate) or delta(mydelta), the SegmentStateBackup will receive calls to onCheckpoint(mystate) and onDelta(mydelta) respectively. For the LOSSLESS resilience mode, the SegmentStateBackup will receive calls to onMediation for every mediation request processed by the master.

For a simple segment service that uses either no resilience or only simple calls to checkpoint, the application can be used without providing a SegmentStateBackup implementation. However, for deltas to work the developer must implement SegmentStateBackup.onDelta to merge the deltas into a complete copy of the master's state. The method SegmentStateBackup.onCheckpoint should also be implemented.

The same applies to an LPP client gateway: for no resilience of for simple checkpointing, no LppStateBackup class is required; for deltas, LppStateBackup.onDelta should be implemented.

For lossless resilience, a SegmentStateBackup is required that implements onMediation. A common pattern is to rely solely on onMediation and to ignore delta() and checkpoint at the SegmentStateBackup.

However, for the LppStateBackup, the delta and checkpoint methods are all that is available. This is because LPPs often change state due to external events that the Monterey framework is not informed of (e.g. the LPP could have a HTTP server, which a new user connects to).

3.6.1   Dynamic Segments

Segments can be added and queried at runtime. The ClientGatewayContext and SegmentServiceContext each provide access to a SegmentSupport instance:

public interface SegmentSupport {
    Map<String, SegmentSummary> getSegments();
    void addSegment(String segment, Callback callback);
}

The getSegments() method allows the code executing at the LPP and Mediator to retrieve the names and descriptions of all segments. The addSegment method allows a new segment to be created at runtime, where the callback is notified on successful creation of the segment.

Example use-cases of dynamic segments include a chatroom application that allows new topics of conversation to be started, or an on-line gaming application where each game is represented by its own segment.

3.6.2   User Interaction

The Monterey platform does not prescribe how client applications should connect to the gateways, however two straightforward methods are suggested:

• Embedded web-server (or other server) at the gateway
  • Gateway embeds a web-server that serves up a Java servlet. Servlet implementation is Monterey-aware, and translates client (web) interaction into Monterey mediation requests.
  • Clients talk to the gateway (LPP) via HTTP
  • Gateway receives Monterey responses and broadcasts as normal, and updates its internal model accordingly. This is communicated back to (web) clients when they next request the page, or asynchronously in real-time using a mechanism such as AJAX.
  • Optionally generated by the New Monterey Classic Project wizard (above).
• A graphical UI hosted at each gateway
  • Using the feature-rich Eclipse Rich Client Platform (RCP) framework.
  • Consult the Eclipse documentation for more information.

3.6.3   Testing and Bots

Bots can be used by a developer to easily simulate multiple users generating load in the system. The bots' workrates are controlled throught the Bots View in the Management Console. This drives the bot code, which can be run on LPPs in the network. Example uses of bots include:

• Stress testing a Monterey Service
• Simulating load to aid the testing of policies
• Generating "test requests" to periodically verify that the Monterey Service is behaving correctly.

Of course, another approach to testing and generating load is to use an external tool (e.g. JMeter) to simulate external clients. The developer is free to choose which is easiest and most appropriate for their use-case.

public class MyExampleBot implements ServiceBeanBot {
    private MyExampleServiceLocator locator;

    public void init(MontereyNetworkEndpoint endpoint) {
        locator = new MyExampleServiceLocator(endpoint);
        locator.setServiceCallsTimeoutInMillis(30*1000);
    }

    public void shutdown() {
        // no-op
    }

    public void doRequest(String segment) {
        MyExampleService service = locator.getService(segment);
        service.myAction();
    }
}

addBot(new MyExampleBot());

makeMediationRequest(BotSegmentBehaviour segment, AppCustomBotParametersT appCustomBotParameters)

getContext().sendMediationRequest()

public class MyExampleBot extends AbstractCdmUserBot<MyExampleCustomBotParameters, MyExampleSegmentCustomBotParameters>{

    @Override
    public void makeMediationRequest(BotSegmentBehaviour<MyExampleSegmentCustomBotParameters> segment, MyExampleCustomBotParameters appCustomBotParameters) throws InterruptedException {
        context.sendMediationRequest(botName, segment.getSegmentId(), "myexample");
    }

    public void onPrivateMessage(String userRef, String segment, String data) {
        // TODO Any correlation logic for bot's workload goes here
    }

    public void onPublicMessage(String segment, String data) {
        // TODO Any correlation logic for bot's workload goes here
    }
}

public class MyExampleLpp implements LppClientGateway, LppClientGateway.ManagementExtension {

    private final Collection<MyExampleBot> bots = new ArrayList<MyExampleBot>();

    public void initialize(ClientGatewayContext context, Object stateToResume) {
        // TODO other lpp logic goes here
        MyExampleBot bot = new MyExampleBot();
        bot.initializeBot("bot1", this, context);
        bots.add(bot);
    }

    public Serializable shutdown() {
        // TODO other lpp logic goes here
        for (MyExampleBot bot : bots) {
            bot.stopTrading();
        }
        return null;
    }

    public void onPrivateMessage(String userRef, String segment, Object data) {
        // TODO other lpp logic goes here
        for (MyExampleBot bot : bots) {
            bot.onPrivateMessage(userRef, segment, (String)data);
        }
    }

    public void onPublicMessage(String segment, Object data) {
        // TODO other lpp logic goes here
        for (MyExampleBot bot : bots) {
            bot.onPublicMessage(segment, (String)data);
        }
    }

    public void onManagementMessage(Object data) {
        // TODO other lpp logic goes here
        if (data instanceof BotBehaviour<?,?>) {
            for (MyExampleBot bot : bots) {
                bot.setBotParameters((BotBehaviour<?,?>) data);
            }
        } else if (data instanceof DemoClock) {
            for (MyExampleBot bot : bots) {
                bot.setClock((DemoClock) data);
            }
        }
    }
}

public static class MyExampleCustomBotParameters {}

public static class MyExampleSegmentCustomBotParameters {}

public interface BotSetup {
    void initialize(BotSet botSet);
    Serializable getAppCustomBotParameter(String botId);
    Serializable getAppCustomBotSegmentParameter(String botId, String segment);
}

bots {
    setup "com.example.mybotsetupfactory.servicename"
}

3.6.4   Custom Provisioning Actions

This section describes how to execute custom actions during the provisioning of network nodes. This enables users to perform setup before the monterey node is created. Possible examples include but are not limited to setting up a VPN or installing security certificates.

Creation of a network node involves 3 steps:

1. Launching a Machine
2. Calling custom provisioning actions
3. Creating the Monterey Network Node

Supported actions include copying files from the management node to the network node and executing commands on the network node.

The API provided is defined by the following Java interfaces:

NodeProvisioningListener

the listener interface which, when implemented, carries out the custom actions

MontereyLocation

can be used to determine if the listener should perform or skip its logic in certain environments.

Host

the host on which to copy files or execute commands

RemoteExecutionResult

encapsulates the result of the command executed

Please view the Javadoc for these interfaces for a complete description of the methods provided by the API.

The following example describes the steps involved in creating and deploying a simple listener that touches a file in the /tmp directory.

1. Create a new plugin project with a MANIFEST.MF file containing the following:

   Bundle-SymbolicName: com.cloudsoftcorp.listener.example
   Require-Bundle: com.cloudsoftcorp.monterey.location,
                   com.cloudsoftcorp.monterey.management
   

2. Create a new class in package com.cloudsoftcorpcom.listener.sample:

   public class ExampleListener implements NodeProvisioningListener {
   
     private static final Logger LOG = Logger.getLogger(ExampleListener.class.getName());
   
     private final List<String> providers = Arrays.asList("aws-ec2","gogrid");
   
     @Override
     public void postCreateHost(Host host, MontereyLocation location) {
   
       final String provider = location.getProvider();
   
       if (!providers.contains(provider)) {
         LOG.info("Skipping Listener: Host:"+host+"; provider:"+provider);
         return;
       }
   
       try {
         RemoteExecutionResult result = host.executeSsh(Arrays.asList("touch /tmp/example.txt"));
         if( result.getExitValue() != 0 ) {
           LOG.log(Level.WARNING, "Error touching file:"+result );
         }
       } catch (IOException e) {
         LOG.log(Level.WARNING,"Could not touch tmp file",e);
         return;
       }
     }
   }
   

   In the example above the listener actions are only performed if the provider is one of 'aws-ec2' or 'gogrid'.

   Please refer to Creating Cloud Accounts section for details of the various providers that are available.

For private clouds where a custom locations conf file is supplied, the author specifies the provider name. The value in the configuration file will be available by calling MontereyLocation.getProvider()

The line that carries out the command is host.executeSsh(Arrays.asList("touch /tmp/example.txt")); The RemoteExecutionResult interface contains an exitValue property, which is inspected for success. See the Javadoc for RemoteExecutionResult for additional information on how this can be used.

1. Create an OSGi service definition for the listener using the File->New->Other->Component Definition menu item. The resulting component.xml file should look like this:

   <scr:component xmlns:scr="http://www.osgi.org/xmlns/scr/v1.1.0"
                  name="com.cloudsoftcorpcom.listener.sample">
     <implementation class="com.cloudsoftcorpcom.listener.sample.ExampleListener"/>
     <service>
       <provide interface="com.cloudsoftcorp.monterey.provisioning.api.NodeProvisioningListener"/>
     </service>
   </scr:component>
   

   The name of the component must be globally unique.

2. The listener bundle needs to be activated on the management node. Update the application deployment descriptor file as shown:

   bundles {
     application {
       ...
     }
     management {
       activate "com.cloudsoftcorp.listener.example"
     }
   }
   

3. When deploying the application via the Clouds View or Management Console, the com.cloudsoftcorp.listener.example bundle needs to be included.

   When developing a Monterey application it is useful to test it in the simulator prior to deploying onto real clouds. On the simulator it is not possible to execute remote commands, and as such the host field in postCreateHost(Host host, MontereyLocation location) will be null. However, you will see log messages if your listener component and bundle have been correctly deployed, giving you confidence that the listener will trigger when deployed to other cloud environments.

4.2.1   Creating Cloud Environments

New cloud environments can be added: right-click in the background of the Clouds view and select "New Cloud Environment...". Alternatively, select File -> New -> Other, then under Cloudsoft Monterey select Cloud Environment. This opens a wizard.

The first page of the wizard asks about the desired cloud providers and locations. Each top-level entry in the tree corresponds to an account with a particular cloud provider (or private cloud). The list of accounts can be edited by clicking "Manage Cloud Accounts...". In the tree, under each account there is a list of available locations. Some or all of these can be selected. This selection will control where Monterey networks can run.

The second page asks about the management plane's web-api credentials. The management console will subsequently use these credentials when connecting to a management node. For further descriptions, including those of the "client" and "admin" roles, see the documentation for "web-users.conf".

An existing cloud environment can be edited: right-click on it in the Clouds view and select edit. However, changes will not take effect in running networks.

4.2.2   Creating Cloud Accounts

A cloud account gives credentials and other configuration information for a given cloud provider or private cloud.

New cloud accounts can be added: right-click in the background of the Clouds view and select "Manage Cloud Account...". Alternatively, select File -> New -> Other, then under Cloudsoft Monterey select Cloud Account. This opens a wizard.

This wizard allows one to remove existing cloud accounts, edit them, or add new cloud accounts. To remove accounts, select one or more accounts from the list and click "Remove". To edit, select a single account from the list and click "Edit". To add a new account, click "Add".

The "Manage Cloud Account..." wizard also has a quick way of creating a cloud environment for a cloud account. Select one or more cloud accounts, and click "Create Cloud Environment...". This will open the New Cloud Environment wizard with all locations of the selected accounts pre-selected.

When adding a new account, the "New Monterey Cloud Account" wizard will appear. The first wizard page asks for the type of cloud account. Select the desired cloud provider and press Next.

The second page of the New Monterey Cloud Account wizard will prompt for a name - this name will appear in the "New Monterey Cloud Environment" wizard; best practice is therefore to use a name that easily identifies the cloud provider, such as "GoGrid" or "GoGrid (My Name)". The second page also includes settings specific to the cloud provider type which are described below.

4.2.2.1   Cloud Account: Simulator

The easiest way to run a Monterey application is through the Simulator. This is a mock network running on the local machine, which can be used to test applications without using a cloud provider's resources. In the simulator, all messages are sent over the network, so it is a faithful test in many respects apart from scale and throughput.

The configuration for a simulator account, shown above, has the following fields:

• Preferred hostname: a hostname or IP address, or blank; used for all socket communications and also for the web-api. It is useful if there are multiple network interfaces and you want to choose one (e.g. if there is intermittent network connectivity, then using 127.0.0.1 for the simulator is advisable).

• Monterey web-api port: the preferred port that the management node will listen on for its web-api

• Custom locations URL: Specifies additional notional locations to be simulated. The file should be in the format described in the section Location Configuration. At a minimum, replace 'ssh-lab-mode' with 'simulator' and 'ssh.datacenter1' with 'simulator.datacenter1'.

  The field is specified as a URL, so for a file must be of the form file:///<path>.

The simulator locations included out-of-the-box have the following location ids in Monterey:

• simulator.london
• simulator.new_york
• simulator.tokyo
• simulator.paris
• simulator.san_francisco
• simulator.singapore
• simulator.barcelona
• simulator.frankfurt
• simulator.hong_kong

For simulator, Calling MontereyLocation.getProvider() returns "simulator"

4.2.2.2   Cloud Account: Amazon EC2

A cloud account for Amazon EC2 allows an end-user to deploy and run monterey applications on Amazon EC2. An AWS account is required, and these credentials must be supplied.

The cloud account wizard, shown above, is configured with these credentials. It has the following fields:

• Access key: the user's AWS access key ID, which can be obtained from the AWS Account Security Credentials page
• Secret key: the user's AWS secret access key
• Security group: a security group that opens the required ports (see Amazon AWS Integration section for a description of configuring security groups).
• SSH private key file per region: these are X.509 Certificate files, which can be downloaded from the AWS Account Security Credentials page, as above. This consists of a pair of files - the certificate itself, and a private key file. You must download and save both files for each region.

The standard set of Advanced Options is available, as well as the following extra options:

• Monterey image name: the naming convention for Monterey images. These are of the form <owner-id>/com.cloudsoftcorp.monterey/<version-number>
• Monterey image version: the specific version range to use when choosing the AMI; this uses the standard format defined in the OSGi specification; it is recommended to leave this field blank so that the version matching the eclipse install will be used (e.g. in Monterey v3.2, the value would be [3.2,3.3) , which means any version greater than or equal to 3.2 and less than 3.3).
• Monterey image owner: optionally specifies the owner of the image. If specified improves performance of finding and launching images. The cloudsoft image owner is 761990928256
• Instance type: the size of the amazon instance, e.g. m1.small, see AWS Instance Types page
• SSH port: the port to use for ssh and scp

The amazon regions have the following location ids in Monterey:

• amazon.us-west-1
• amazon.us-east-1
• amazon.eu-west-1
• amazon.ap-southeast-1
• amazon.ap-northeast-1

For Amazon, Calling MontereyLocation.getProvider() returns "aws-ec2"

For a more detailed description of the amazon integration, see the Amazon AWS Integration section.

4.2.2.3   Cloud Account: GoGrid

A cloud account for GoGrid allows an end-user to deploy and run monterey applications on GoGrid.

The cloud account wizard, shown above, is configured with the GoGrid credentials. It has the following fields:

• Key: the key, along with the shared secret, can be created/obtained from GoGrid's control panel under My Account > API keys. The key is typically 16 digits, but yours may be different.
• Shared secret, typically 12 digits, created/obtained with the corresponding key.
• SSH public key file: this can be any ssh public key file; it will be installed on the GoGrid VMs, so that the corresponding private key can subsequently be used to ssh into any of the VMs created using this cloud account (e.g. using a command like ssh -i /path/to/privatekeyfile root@ipaddr).
• SSH private key file: the private key file that goes with the public key file; this is installed on the management node so that it can ssh into network-node VMs it creates.

The standard set of Advanced Options is available, as well as the following extra options:

• Image id (override): the integer id of a specific GSI to use (note this id isn't readily available from the GoGrid UI); leave blank to allow Monterey to select the latest compatible image.
• Image pattern (override): a regular expression pattern to use to find a GSI to use; leave blank to allow Monterey to select the latest compatible image.
• Cloud server size (1 to 6): the size of the GoGrid instances to use, 1 being the smallest; these numbers correspond to the configurations shown here: GoGrid Cloud Servers page
• SSH port: the port to use for ssh and scp.

The GoGrid data centers (see GoGrid's Data Centers page) have the following location ids in Monterey:

• gogrid.us-west-1
• gogrid.us-east-1

For GoGrid, Calling MontereyLocation.getProvider() returns "gogrid"

4.2.2.4   Cloud Account: VMWare vCloud

Configuring vCloud v1.01:

The cloud account wizard, shown above, has the following fields:

• Endpoint: the URL of the vCloud API; this is normally of the form https://myservername/api where myservername is the hostname or IP of the vCloud Director
• Default Network: name of the default network, in the default VDC that your vApp will join
• vApp Template file name pattern: This will be used to find the template file. It is matched against file names and the first match is used.
• Username and Password
• SSH public key file: this can be any ssh public key file; it will be installed on the vCloud VMs, so that the corresponding private key can subsequently be used to ssh into any of the VMs created using this cloud account (e.g. using a command like ssh -i /path/to/privatekeyfile root@ipaddr).
• SSH private key file: the private key file that goes with the public key file; this is installed on the management node so that it can ssh into network-node VMs it creates.
• Custom locations URL: Specifies the location of this vCloud. The file should be in the format described in the section Location Configuration.

The standard set of Advanced Options is available.

For vCloud, Calling MontereyLocation.getProvider() returns "vcloud"

4.2.2.5   Cloud Account: SSH Lab-mode

SSH Lab-mode will use a fixed pool of machines when provisioning and releasing Monterey management and network nodes. This same pool can be shared by multiple Monterey networks, from multiple users.

The list of machines is stored and maintained within a google spreadsheet. The spreadsheet should have the following columns:

• owner: this field is auto-populated when a machine is claimed by a Monterey network.
• lastChangeTime: this field is auto-populated, it is updated each time the machine is allocated or released.
• publicAddress: the public IP address (or hostname) for this machine; blank if none. When creating management nodes, machines with public addresses will be chosen so that Monterey Management Consoles or other monitoring tools can access them.
• privateAddress: the IP address (or hostname) within that subnet, if different from the public address; blank if none.
• managementNodeHomeDir: the absolute path to the installation directory of monterey-management-node. For example: /home/monterey/monterey-management-node/
• networkNodeHomeDir: the absolute path to the installation directory of monterey-network-node. For example: /home/monterey/monterey-network-node/
• sshPort: specifies the TCP port to use when logging-into the machine over SSH; or blank to use default 22.

The spreadsheet should have at most one row per publicAddress.

Note, if using a VPN to provide access, all addresses should be considered "public" because the VPN is ensuring that all relevant machines can communicate.

Note, the private address field is useful when some machines do not have a public address, but where there is a private address visible from within that network. In such a configuration, management nodes will always be placed on machines with public addresses; network nodes can use any machine. However, if the machine is not visible from outside the subnet then it cannot be used in an inter-cloud network - a solution is to use a VPN so that the same group of IP addresses or hostnames can be used throughout. Also note that if a machine has a different IP address from within the subnet than from outside, then it is recommended to configure DNS such that the hostname resolves correctly and then to use hostnames throughout.

The cloud account wizard, shown above, is configured with this google spreadsheet's name and the name of the worksheet within it. A google username (e.g. my.name@google.com) and password are also needed to access the spreadsheet - the account must have read/write permission.

Management and network nodes will be started and stopped by invoking scripts over ssh. The default ssh credentials (i.e. blank) will work if the user running studio has authorized_keys set up to be able to ssh in without requiring a password. Alternatively, the username to use can be set along with a particular private key file. For example, entering username myname and keyfile id_dsa will use the command::

ssh -i /home/myname/.ssh/id_dsa myname@<hostname>

The custom locations URL defines the location in which these machines will run. The URL should refer to a locations configuration file (e.g. file:///my/path/locations-datacenter1.conf). The file should be in the format described in the section Location Configuration.

The standard advanced options are available:

For Ssh Lab Mode, Calling MontereyLocation.getProvider() returns "ssh-lab-mode"

4.2.3   Advanced Options

A common set of advanced options is available to customise the operation or usage of most types of cloud account. Any additional advanced options are described in the main section for each account type. The Appendix has further details on the Management Web API configuration options.

The standard advanced options are as follows:

• Monterey node port: the TCP port for the management node or network node to use for communication between monterey nodes.
• Monterey web api port: the preferred port that the management node will listen on for its web-api; if this port is in use then the port number will be incremented until an available port is found.
• Monterey web api protocol: the protocol that the management node will use for its web-api; this must be either http or https.
• Monterey web api SSL keystore: a JKS keystore file containing a provate key and certificate for use by the web-api; this must be provided when using the https protocol.
• Monterey web api SSL keystore password: the password used by the web-api to access the JKS keystore.
• Monterey web api SSL key password: the password used by the web-api to access the private key in the JKS keystore.
• Monterey hub lpp port: the port to use for external proxy LPP connections.
• Logging file override: a logging.properties file to be used by the management and network nodes, instead of the default.

The OK button will be disabled until valid options are entered in all fields.

4.2.4   Application deployment

You can drag-and-drop Monterey service projects onto cloud environments within the Clouds view. You can also drag-and-drop OSGi bundles (from within Eclipse or from the file system), or drag-and-drop a application descriptor file.

This will bring up a Deploy Monterey Application dialog. This allows you to select an application descriptor file, the projects to deploy, and any additional OSGi bundles to deploy. The dialog will be pre-populated with whatever was dragged onto it.

The drop-down list of application descriptor files is auto-populated with the configuration files found in the selected projects and bundles. Alternatively, you can browse the file system to select your descriptor file.

The projects listed will be deployed to the Monterey network. They will be automatically exported to generate OSGi bundles, and should therefore be Eclipse plugin-projects. The list of pre-built OSGi bundles will also be deployed. The projects and pre-built OSGi bundles should together contain the segment-service code, as well as any LPP code required.

To deploy, click Ok. This will launch a new Monterey network. A corresponding new entry will appear under the cloud environment.

Alternatively, you can open the deploy dialog by right-clicking on the cloud environment and selecting Deploy Application....

4.2.5   Manually generating OSGi bundles

Instead of selecting projects to deploy, it is possible to manually generate the OSGi bundles. The standard deployment dialog can still be used, but selecting the pre-built OSGi bundles rather than the projects.

To create a bundle from your project:

1. Right-click on the project in Eclipse's Project Explorer, and select Export...
2. Expand the Plug-in Development group and select Deployable plug-ins and fragments, and click Next. ("plug-in" is Eclipse's terminology for an OSGi bundle)
3. Next to Directory, click Browse and choose a location to save the bundles. Eclipse will create a folder called plugins as a child of your selected directory, and place the bundles in it.
4. Click Finish.

4.2.6   Shutting down networks and Removing them from the Clouds View

To shut down a network, right-click on it and select Shutdown network. Note that you will not be able to do this if there are any running nodes in the selected network - you must first launch a Monterey Management Console and release all nodes in the network, before shutting down.

You can also remove a network from the clouds view without shutting it down or otherwise affecting it. Right-click and select Remove from list.

4.4.1   Standalone Monterey Management Console

The standalone Monterey Management Console allows you to connect to a running management node, to view and manage that network.

The Resources View is the default view when opening the control pane. It shows a hierarchical view of the available resource locations, including all allocated nodes in the network. It includes the status and runtime details about each node:

The Nodes View shows the same resources in a sortable "flat" table, with additional details about runtime performance metrics:

Both views provide identical actions for provisioning and rolling-out nodes (see below), via their respective toolbar menus and right-click context menus.

4.4.2   Provisioning and Rolling Out

The first step in creating a network is to provision some nodes. Right clicking on a location or click on the view's drop-down menu. From here you can select where to provision new nodes.

You can choose to provision a custom number of nodes, or use the quick setup in the menu. Each node is associated with a specific location, and this cannot be changed after allocation.

After a successful provisioning action, you can begin to use the nodes by rolling out the Monterey software. When allocated, the nodes are normally in a spare state, without a specific role within the network. Select All the nodes you wish to deploy to, and right click to bring up the context menu. The Roll-out option is now enabled and from here it is possible to either roll out a pre-set configuration, or set custom parameters.

There are four standard Node types: LPP (the gateway), MR (an internal router), M (the mediation node), and TP (an internal proxy). A typical network will have an equal number of each node type. If satellite/hubs are enabled (using bots { satellites true }), you can choose to roll out bots as well (these additional LPPs contain any client specific code in your gateway classes). Once you have rolled out to the spare nodes, they will appear in the network topology view.

Tip

It is is possible to quickly provision and roll-out a number of preset configurations. This can be done using the Quick Setup menu from the Resources View toolbar or right-click menu:

4.4.3   Segments and the Segment List View

The Segment List View shows the current segments defined for the network. When the network is active, the per-segment statistics and run-time information can also be seen in the segment list. Each segment has an unique identifier, normally a short string; this is the only necessary property for a segment. Segments can be customized with a more descriptive name, and textual tags. Additionally, a segment has a color attached, used in the various graphs and charts throughout the management console. Colors can be set for a range of segments, using the context menu.

It is also possible to do bulk operations on segments here by selecting the segments and using the context menu; for example migrating a group of segments to one particular Mediator.

4.4.4   Geographies

Each node resides in a location. For the localhost provisioner these locations are notional, but for other providers the location will be used to distinguish between nodes in different "regions" of a cloud provider (using Amazon parlance), or data centers for a private cloud. or data centers.

The geography view shows the activity of nodes in each location, where locations are normally at the granularity of data centers (or clusters of data centers, such as Amazon's us-west-1). A circle on the map represents each location where there are nodes deployed. The area of the circle is indicative of the number of nodes deployed, and the color of the circle indicates the health of those nodes.

There is also a shadow cast upon the geography view, indicating the current Monterey clock time. This clock is used by some bots to adjust activity levels. You can change the current Monterey clock by dragging the shadow left or right to change the time of day. The current speed the clock runs at can also be changed here from the toolbar menu. More about the clock and bots will be covered further on.

You can use the geography view to filter the current graphs and views. Clicking on a circle will only display information for that specific location, and the titles of the views and graphs affected will change to reflect this. Clicking on the background will reset the views to showing all nodes as before.

Finally, when running in the simulator, it is possible to simulate latencies between locations. Using the Simulated Latency page, you can configure the delay in milliseconds between locations. When these values are changed, the new setting is used for all newly provisioned nodes; existing nodes will use the earlier settings.

4.4.5   Policies

The management console allows you to manually change the network topology and the location of segments, as well as allocate and deallocate nodes. In a real-world deployment, however, this should be done automatically, responding to changes in messaging volume, per-segment demand, regional activity shifts, and other factors.

Monterey Middleware supports custom management policies so that changes are made in response to these factors and according to the priorities (and budgets) of individual projects, to ensure the desired quality of service level throughout the network.

Monterey includes illustrative policies, which capture the following use cases:

• "Follow the Sun" (optimise for latency): relocates segment data and processing to follow the locus of of active users, minimising average latency to the majority of customers. The policy detects high message rates in any given location for a given segment and will migrate the segment to the statistically busiest location.

• "Follow the Moon" (optimise for cost): segments can be relocated to reflect changes in price linked to running the segment in a given location, with the goal to minimise costs. The policy can be kept appraised of relevant current prices, relocating segments to the cheapest available location.

• Watermark Elasticity (optimise for usage): rolls out, reverts and re-routes nodes at a given location to optimise resource usage. High- and low-watermark settings guide migration and re-routing decisions, so that the policy keeps all nodes reasonably active (for resource efficiency) without any over-active (bottleneck) nodes. The policy will scale-out and scale-back from the pool of spare nodes, and can change the type of nodes based on the current usage pattern. This policy can be used in combination with the Node Pool Elasticity policy.

• Node Pool Elasticity (optimise for usage): an auxiliary policy that grows and shrinks the number of nodes in a given location, based on requests for more nodes that are made by other policies.

  The policy will scale-out (request new resources) and scale-back (give resources back) as needed and as permitted by the public or private clouds being used.

These policies can cut latency (and thus significantly improve the end-user experience) and reduce costs dramatically. This is true for globally-deployed applications, and also for applications in a private cloud where scaling out scaling back is desired.

Policies are shown and managed in the Policy View. To add a policy, right-click on the background of the table in this View. Right-clicking on any selected policy will allow you to suspend, resume, or remove the policy.

Some policies are available at different scopes, either regional or global. Regional policies are restricted to a specific location, while global policies work on the entire network. A common configuration is to enable the Watermark Elasticity policy in each region (e.g. doing load-balancing and scaling within a data-centre) and the Inter-Geography policy on the global level (it has no effect if it is applied to a single location).

Within the Preferences pane, operating parameters for these policies can be adjusted under Monterey Middleware -> Policy.

The Watermark Elasticity can be scaled to deal with less powerful machines or more computationally-intensive applications (numbers less than 1 decrease the watermarks).

The follow-the-sun policy has many more parameters for fine-tuning; assistance on each is available in the product in tool-tips which appear when the mouse hovers over each:

• Percentage of total: There are multiple triggers which must all be satisfied in order to move a segment X to a geography A. One trigger looks at the percentage of requests in X originating from A. Mathematically, where x is total number of requests submitted in X across the Monterey network, and x_A is number of requests from geography A, and where A is the most prolific geography (arbitrarily chosen in case of ties, so it is recommended to specify at least one majority parameter as well); this parameter T defines a number in [0,1] such that x_A > T*x + T' in order for X to be migrated to A. (T' is defined below as the 'delta above'.)
• Delta Msgs/Sec above Percentage of Total: There are multiple triggers which must all be satisfied in order to move a segment X to a geography A. One trigger looks at the percentage of requests in X originating from A. Mathematically, where x is total number of requests submitted in X across the Monterey network, and x_A is number of requests from geography A, and where A is the most prolific geography (arbitrarily chosen in case of ties, so it is recommended to specify at least one majority parameter as well); this parameter T' defines a number such that x_A > T*x + T' in order for X to be migrated to A. (T is defined above as the 'percentage'.)
• Absolute Msgs/Sec: There are multiple triggers which must all be satisfied in order to move a segment X to a geography A. One trigger looks at the total number of requests in X originating from A. Mathematically, where x is total number of requests submitted in X across the Monterey network, and x_A is number of requests from geography A, and where A is the most prolific geography (arbitrarily chosen in case of ties, so it is recommended to specify at least one majority parameter as well); this parameter T defines a non-negative number such that x_A > T in order for X to be migrated to A.
• Percentage Majority: There are multiple triggers which must all be satisfied in order to move a segment X to a geography A. One trigger looks at the percentage of requests in X originating from A as compared against other geographies. Mathematically, where x is total number of requests submitted in X across the Monterey network, and x_A is number of requests from geography A and x_B from B, and where A and B are the most prolific geographies with x_A >= x_B, this parameter T defines a number in [0,1] such that x_A - x_B > T*x + T' in order for X to be migrated to A. (T' is defined below as the 'delta above'.)
• Delta Msgs/Sec above Percentage Majority: There are multiple triggers which must all be satisfied in order to move a segment X to a geography A. One trigger looks at the percentage of requests in X originating from A as compared against other geographies. Mathematically, where x is total number of requests submitted in X across the Monterey network, and x_A is number of requests from geography A and x_B from B, and where A and B are the most prolific geographies with x_A >= x_B, this parameter T' defines a number such that x_A - x_B > T*x + T' in order for X to be migrated to A. (T is defined above as the 'percentage'.)
• Msgs/Sec Majority: There are multiple triggers which must all be satisfied in order to move a segment X to a geography A. One trigger looks at the number of requests in X originating from A as compared against other geographies. Mathematically, where x is total number of requests submitted in X across the Monterey network, and x_A is number of requests from geography A and x_B from B, and where A and B are the most prolific geographies with x_A >= x_B, this parameter T defines a non-negative number such that x_A - x_B > T in order for X to be migrated to A.

Follow-the-moon policies can be configured using the Costs Per Location preferences. This allows a cost function to be defined for each location. For example, the function below specifies that between 8am and 6pm the cost is 0.12 per hour, and between 6pm and 8am the cost is 0.06 per hour:

8-18h @ 0.12, 18-8h @ 0.06

It is also possible to exclude a given location from all inter-geography migrations. Segments will never be migrated into or out of excluded locations by follow-the-sun or follow-the-moon policies. This feature is accessible within the preference pane.

Node Pool Elasticity can be configured using the following parameters, which control when nodes should be released:

• Min hot-spares retained: The minimum number of spare nodes that should be kept in each location. Spare nodes will only be released when there are more than this number of nodes available.
• Minimum inactivity period: The length of time in milliseconds that the spare nodes must exist for before they can be thought of as "unused". Any activity, such as a policy requesting additional nodes, will zero the inactivity count.
• Node release rate/sec: When releasing unused nodes, the maximum number of nodes that can be released each second. For example, 0.1 mean one unused node can be released every 10 seconds.

Custom policies can also be loaded and controlled through the Policies View. This allows a richer semantics of jurisdiction requirements or bespoke optimization strategies. For more information on creating a custom policy and initializing it, please consult the Appendix.

4.4.6   View Reference

The basic management console views are:

• Lists:
  • Resources: Displays all nodes, arranged hierarchically based on location; allows to provision, release and roll out new nodes, as well as quick setup for rolling out an entire monterey network.
  • Nodes: Lists all nodes and their status.
  • Segments: Allows the manual control of segment migration, the saving and loading of segments, as well as the configuration of custom colours for the display of segments in other views.
  • Bots: Provides access to Bot scripting, the adjustment of bot activity weighting both over time and towards a individual segment or segment group, as well visualisation of the current activity weighting.
  • Policies: Used to configure automatic migration policies such as Watermark Elasticity and various Inter-Geographical policies.
• Visualization:
  • Network Topology: Displays the logical topology of rolled-out nodes and displays migrations as they happen.
  • Geographies: Indicates the volume of messaging in geographical areas by the size of the circles. Also, allows the filtering of the other views by clicking the geographical location as well as the manual adjustment of the simulation time.
• Bar Charts:
  • Service/Client Segment Activity: Dynamic bar-graph of the volume of messaging broken down between nodes and locations. Also displays the watermark (see watermark policies) and breaks down each volume by segment (configure segement colours for increased clarity).
• Line Graphs:
  • Mediation/Total Message Rate: Trace-graph of the messaging rates, broken down into requests, responses and broadcasts.
  • CPU Resource: Trace-graph of the available, rolled-out and utilised CPU resources.
  • Round-trip Time: Trace-graph of the maximum and mean round-trip times.
• Other:
  • Cost Evaluation: Provides an illustrative example of the potential monetary savings the follow-the-moon policy can offer against other strategies

All the basic views provide filtering (most support locational filtering and node type, where appropriate). You can also change the displayed columns in list views; both are accessible via the dropdown menu accessed via the arrow in the top right of a view area.

4.4.7   Deploying Applications

The management console supports deploying and undeploying applications. This is done through the Deployment menu.

Other options in the Deployment menu are "Describe application" for finding out about the currently deployed application, and "Reset management console". The latter will reconnect to the management node and re-initialize the management console's views (note that in simulator mode this has the side effect of stopping the entire simulated network).

See Deploying to a Monterey Network for further information on deployment.

4.4.8   Preferences

The Monterey Management Console has a preferences pane for setting various configuration options.

For options that were set at startup, these are shown as immutable values in the preferences pane. For example, the "control plane location" is set automatically based on the way that the Management Console was started (e.g. the simulator always uses "Simulator (New Localhost Network)", whereas the standalone Management Console connects by default to a remote control plane over HTTP).

When connecting to an existing management plane, the preferences pane include the URL of a management node and the credentials to use when connecting.

There is an option to "force local IP", which controls the IP address used by the management console. This is useful when there are multiple network adapters.

"Workrate reporting period" controls how frequently the management console will retrieve network metrics, and thus how frequently the UI will refresh.

Under Monterey Middleware is the preferences page Policy that deals with policy settings. See Policies.

4.6.1   Monterey Network Setup

A Monterey network is configured with several network nodes and a single management node.

The developer/deployer interacts with the Monterey network via the Management Console running on their local workstation, connected to the management node over HTTP. To provision a node for use within the Monterey network, the management node connects to it over SSH and runs the appropriate scripts that were installed when the machine was set up (see below). Only the management node is required to be addressable from outside the network (i.e. from the developer/deployer's workstation).

The unpacked Monterey Studio download package contains archives for installation on the machines (which may be "bare-metal" hardware or VMs) that will comprise the Monterey network infrastructure.

4.6.2   Setting Up the Network Nodes

It is best practice to run a single network node per (virtual) machine.

1. Configure each machine according to the common software installation described in the Appendix.
2. Locate the Monterey-NetworkNode-[version].zip in the installs directory of the unpacked Monterey Studio installation, and copy to each of the machines that will be the network nodes.
3. Unpack the archive to a location that is visible to the machine's monterey user, such as the ~monterey/ home directory, or a suitable shared location such as /opt/monterey/. The archive unpacks to a single directory called monterey-network-node/.
4. Install custom OSGi bundles for your applications as needed in the osgi/plugins sub-directory.

4.6.3   Setting Up the Management Node

1. Configure the machine according to the common software installation described in the Appendix.
2. Locate the Monterey-ManagementNode-[version].zip in the MontereyNetworkSetup directory of the unpacked Monterey Studio installation, and copy to the machine that will be the management node.
3. Unpack the archive to a location that is visible to the machine's monterey user, such as the ~monterey/ home directory, or a suitable shared location such as /opt/monterey/. The archive unpacks to a single directory called monterey-management-node/.
4. Install custom OSGi bundles for your applications as needed in the osgi/plugins sub-directory.
5. Configure the management node by editing the files below in the conf directory.
6. A monterey network can be launched by creating and using a ssh lab mode cloud account. Alternatively, the management node can be controlled via the scripts in the monterey-management-node/scripts/ directory.

managementBusBundle=com.cloudsoftcorp.monterey.activemq
managementBusCommsFactory=com.cloudsoftcorp.monterey.activemq.comms.ActiveMqPubSubCommsFactory
managementBusAddress=tcp://management-node:61616 broker-token

./start_activemq_broker.sh broker:tcp://management-node:61616/

4.6.4   Deploying to a Monterey Network (Deprecated)

Monterey applications can be deployed to distributed Monterey networks as easily as running them in the simulator. The recommended way is to use the clouds view, such as the ssh lab mode.

The following approach is deprecated:

Before deploying, the uninitialized network should be available, with a management node running able to provision network nodes, and the OSGi bundles for your application installed on all nodes. Then, to deploy, create a Monterey Deployment launch configuration, just as was done previously in the Monterey Simulator section: the only difference is that along with the application deployment descriptor file, the configuration requires the details for accessing a management node in the target Monterey network.

If there is already an application running at that management node, then an error dialog will be displayed saying that deployment failed. Only one application can be deployed on a given management node at a time.

Undeploying and deploying applications can be done through the "Deployment" menu of the management console. To undeploy an existing application, there must first be no nodes running. The "undeploy" action will then remove the application from the management node.

The "Deploy application" menu option allows an application to be deployed to the management node. The user selects a deployment descriptor file, which is then sent to the management console. If an application is already deployed, this action will fail.

path/to/MontereyManagementConsole \
  -deploy
  application=path/to/DeploymentDescriptor.conf \
  bundles=path/to/bundle1.jar,path/to/bundle2.jar \
  management.url=http://monterey.example.com:8080/ \
  management.user=username \
  management.password=password \
  -noSplash

path/to/MontereyManagementConsole.app/Contents/MacOS/MontereyManagementConsole \
  -deploy
  application=path/to/DeploymentDescriptor.conf \
  bundles=path/to/bundle1.jar,path/to/bundle2.jar \
  management.url=http://monterey.example.com:8080/ \
  management.user=username \
  management.password=password \
  -noSplash

path/to/MontereyManagementConsole \
  -undeploy
  management.url=http://monterey.example.com:8080/ \
  management.user=username \
  management.password=password \
  -noSplash

4.8.1   Hot Code Swapping

Hot code swapping describes the situation where it is necessary to update code which is running without interrupting service. In many other platforms this requires extra implementation or special facilities, if it is supported at all. However using application mobility provided by Monterey it is possible to update live code seamlessly.

As with ordinary migration, the current mediator is told to shutdown, and the state sent to a new mediator on the network. Unlike a normal migration, however, the new mediator is running a newer version of the code. This hot code swap is transparent to the clients, just as with ordinary migration, and the sequential ordering of messages is preserved.

4.8.2   Security

Security at the communications layer in Monterey Middleware can be ensured by using either HMAC-SHA-1 authentication (for each message) or SSL certificates and encryption (for each connection), depending on requirements. This can be applied on a per-path basis. It is also possible to configure Monterey to use a custom transport (such as a messaging system or IPsec VPN) to comply with site-specific requirements or for additional security. This can be done in conjunction with HMAC or SSL. The management node can be supplied with a JKS keystore containing an SSL certificate, in order to secure Management Web API communications using HTTPS instead of the default, HTTP.

At the application layer Monterey can enforce Java permissions; this is principally useful when the code running on the Monterey is potentially untrusted.

At the infrastructure level, Monterey can be locked down by firewall or iptables rules such that only the LPP client nodes are externally addressable, and such that these nodes are only accessible on the ports used by the application-specific client gateway code. For example, with an application whose client gateway launches a web server on port 8080, Monterey can run on a private subnet with an external firewall forwarding traffic to the web servers only. Communications within the Monterey network uses only non-root ports numbered 1025 or higher; it is not necessary to enable any services on lower numbered ports. It is not necessary that the Monterey nodes be able to contact the internet, unless the application-specific LppClientGateway and MediationSegmentService require it.

By default, the Monterey nodes and Management Console will open a port to accept incoming connections from a remote OSGi console, which can be very useful when diagnosing problems with OSGi bundles at runtime. This is achieved by specifying the -console <port> argument in the startup scripts. If desired for security reasons, this can be removed without affecting the operation of the network or deployed application.

4.9.1   Requirements

An AWS account is required to deploy and run on Amazon AWS. Appropriate credentials must be supplied prior to deploying the application. The following Amazon security information is required:

1. A key pair for each region. We advise that you name each key pair monterey.region - for example, monterey.us-west-1. You must also download and save the private key file for each key pair. Key pairs are managed on the Amazon EC2 Management Console Key Pairs page.
2. The user's AWS access key ID and secret access key, which can be obtained from the AWS Account Security Credentials page
3. X.509 Certificate files, which can be downloaded from the Security Credentials page, as above. This consists of a pair of files - the certificate itself, and a private key file. You must download and save both files.
4. The name of a security group that has all of the required ports opened.

4.9.2   Configuring a security group

The easiest way to setup a security group is to use the Amazon EC2 API Tools.

You will need to install the Amazon EC2 API Tools, which can be downloaded here.

Next, configure the EC2 API Tools. These commands will work for bash and similar shells:

export AWS_ACCESS_KEY_ID=access_key_id_from_step_2
export AWS_SECRET_ACCESS_KEY=secret_access_key_from_step_2
export EC2_CERT=certificate_file_from_step_3
export EC2_PRIVATE_KEY=private_key_file_from_step_3
export EC2_HOME=location_where_ec2_api_tools_are_installed
export PATH=$EC2_HOME/bin:$PATH

Now for each region - eu-west-1, us-east-1, us-west-1, ap-southeast-1 and ap-northeast-1 - create a security group (we recommend calling it monterey) and specify the firewall ports that Monterey requires:

export EC2_REGION=<region>
export EC2_URL=https://$EC2_REGION.ec2.amazonaws.com

ec2-add-group monterey -d "Monterey Middleware"
ec2-authorize monterey -p 22
ec2-authorize monterey -p 8080
ec2-authorize monterey -p 8443
ec2-authorize monterey -p 43500
ec2-authorize monterey -p 43501

This completes the required configuration.

4.9.3   Architecture overview

Applications written in Monterey can be deployed and run on Amazon AWS.

The locations available in Monterey correspond to the Amazon regions: US-East (Northern Virginia), US-West (Northern California), EU-West (Irelend), Asia Pacific-South East (Singapore) and Asia Pacific-North East (Tokyo).

When deploying an application, a VM is automatically provisioned to run a management node. When network nodes are subsequently started, a VM is automatically provisioned for each. Cloudsoft supply AMIs for creating these management and network node VMs.

Interaction with Amazon and the VMs is through a combination of web-service calls and SSH.

The instances created from the Cloudsoft AMIs contain no special SSH configuration. The only way to log into the instance is to supply your private ssh key file. For example:

ssh -i ~/.ssh/id_rsa-myname.us-west-1 ec2-184-72-171-85.compute-1.amazonaws.com

Under the covers, deploying and running an application on AWS involves the following steps:

• To create the management node:
  • launch a new VM in the desired region
  • Configure user-data for the instance, to supply the AWS access key ID and the AWS secret access key
  • Copy the private SSH key files, required to SSH into network-node VMs in each region
  • Copy configuration files containing the user's configuration options
  • Start the management node process.
• On provisioning a network node, the management node will:
  • Launch a new VM in the desired region This uses the supplied AWS access key ID and secret access key credentials, and the keypair name
  • Start the network node process. This is done over SSH, using the supplied SSH private key file

You can view the running VMs in each region by logging on to the AWS Management Console.

VMs will be tagged with three values: a Node Type to indicate whether a VM is a management or a network node, a Version indicating the version of Monterey Middleware they're running (taken from the AMI they were built with) and an optional Network ID which can be provided when launching the management console; the Clouds view automatically populates the Network ID to match the one displayed in the view, while the management node will automatically configure any nodes it provisions with a matching ID.

4.9.4   Adding cloud environment for Amazon EC2

Click on the small arrow at the top right of the The Clouds View and select New Cloud Environment... This will start the New Cloud Environment Wizard. Select Amazon from the list of cloud environment types and click Next.

The Amazon service configuration page contains the following fields:

• Cloud environment name: this is the name of the service that will appear in the Clouds view. For example: "My Amazon EC2 account"
• Management Console credentials: the usernames and passwords for the accessing the management web API. These passwords are encrypted and used to generate the web-users.conf file. The admin role gives full access; users with the client role can only register and unregister external LPPs.
• Amazon AWS credentials Access key and Security key: these are provided by Amazon AWS and can be obtained from the AWS Account Security Credentials page
• Security group: the name of the security group created earlier, such as monterey
• Regions list: for each Amazon EC2 region that you will be using, you need to add the region to this list and identify an SSH private key file. For each region, the corresponding public key file needs to be uploaded to the Amazon EC2 control panel, and named monterey.region.
• Start Management node in: this drop-down list will contain all the regions you defined previously, and identifies in which region the Management node should be started in.

Clicking Finish will then create the new cloud environment for Amazon. The Amazon AWS credentials and other settings will be verified and, if there are no problems, the new cloud environment will appear in the Clouds view. You can now deploy applications to Amazon EC2 as described in The Clouds View.

5.1.1   Running the Application

• Application fails to start

  There are numerous interdependencies any one of which being slightly wrong can cause an application, either Monterey Studio or your own Monterey Application from the management console, not to start. Following are some common causes and tips for resolving this:

  • Ensure that the Java 1.6 (or later) from Sun is installed and accessible.
  • Pass the -clean option on the command line.
  • If you are in a mixed 64- and 32- bit environment, see below.
  • If all else fails, consult the log file (typically in the configuration directory, or available in a console by running Monterey Studio at the command-line with -consoleLog) or re-install.

• Error message: java.lang.UnsatisfiedLinkError: Cannot load 32-bit SWT libraries on 64-bit JVM

  If you are in a mixed 64- and 32- bit environment it may be necessary to force a particular architecture. This can be done by passing the Java VM argument -d32. When launching from within studio, add this to the launch configuration, in the Arguments tab, by adding -d32 as a VM argument. When launching the standalone Management Console, edit the .ini file.

  Within Eclipse, you can add this as a default VM argument for whenever you launch Java: in Preferences, navigate to Java -> Installed JREs, then select your chosen installed JRE and click Edit, then add the Default VM Argument of -d32.

  With OS X it can be particularly complicated because some versions of Apple's Java 1.6 are 64-bit only. Check you are up-to-date with OS X Updates as the newest versions do include both 32-bit and 62-bit JVMs. Installing Java 1.6 supporting 32- and 64-bit can help (although this is not supported by Apple), as can forcing the OSGI arch to be x86_64.

• Error message: java.lang.UnsatisfiedLinkError: Cannot load 64-bit SWT libraries on 32-bit JVM

  If you are using the 64-bit version of Monterey Studio with a 32-bit JVM it may be necessary to force a particular architecture. This can be done by passing the Java VM argument -d64. When launching from within studio, add this to the launch configuration, in the Arguments tab, by adding -d64 as a VM argument.

• On Mac OSX, Java compiler errors: ...only available if source level is 1.5 or: Unbound classpath container: 'Default System Library'

  A recent Apple Java Update modified how Java Virtual Machines are installed and configured, and the current versions of Eclipse are unable to detect the new configuration. Eclipse either falls back to assuming Java 1.4 as the default JVM, or no default JVM at all.

  You must must manually identify the correct JVM that Eclipse or Monterey Studio uses as default. Open the Preferences dialog, and go to the Java -> Installed JREs pane. This should list a number of JVMs from versions 1.4 to 1.6 (or later) - if it does not, click the Search button. Then, ensure that an appropriate JVM is ticked to indicate it is the default JVM - we require a JavaSE 1.6 JVM. Click OK.

  If there are still errors: open Preferences and go to the Java -> Compiler pane. Ensure the Compiler compliance level is set to the same Java version as selected earlier. Click OK. If problems persist, right-click on a project with errors and select Properties. Go to the Java Compiler pane. Ensure that either Enable project specific settings is off, or that the Compiler compliance level is set to the same Java version as selected earlier.

• Error message: Queueing viewer updates

  This message appears in some environments from time to time. This is a known Eclipse bug and does not seem to cause any problems apart from irritation. Restarting the application and doing fewer memory-intensive activities in the application can help. We are actively looking for a solution.

• Monterey Application fails to run locally (in Simulator)

  The Studio contains a Console view which will contain logging information for the simulation run. In some cases, Simulator will cache data from the Monterey Application environment. In the launch configuration dialog there are options to ensure that it cleans the workspace and the configuration area; enabling these may help. The OSGi console is available for additional debugging and exploring on port 8021, for example using the unix command nc 8021.

• Monterey Application fails to run remotely

  Check that the remote network is available and that the OSGi bundles for the Monterey Application have been correctly installed there. OSGi can cache data so it may be necessary to remove the monterey-network-node directory and re-install if the application has been changed. (In some situations, OSGi containers can be started with -clean to clear its cache; consult standard OSGi and Eclipse Equinox references for more information.)

• Monterey Application does not shut down cleanly

  Monterey runs as a distributed application, even if only running on localhost, and while Monterey attempts to progress and orderly shutdown, it may not complete in time (particularly when the management console is launched from within Monterey Studio). Errors or warnings may be displayed, or some non-daemon node threads may be left running (preventing the application from shutting down). In all cases it is safe to forcibly terminate the application from within Eclipse (clicking the red "stop" button) or at the OS level.

• Nodes do not start when using ActiveMQ

  ActiveMQ tries to resolve the node's hostname to an IP address during initialization. This may not be the case on some Linux configurations and/or if the node is not registered in DNS. You can test this using the hostname and ping commands. For example:

  $ hostname
  node-53
  $ ping -c 1 node-53
  PING localhost (127.0.0.1) 56(84) bytes of data.
  64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.010 ms
  ...
  

  If the ping command returns unknown host instead, then you should ensure that the node is correctly registered in DNS or that the hostname is added to /etc/hosts (an address of 127.0.0.1 is acceptable for the hostname and is easier in some virtualization environments).

• NoClassDefFoundException for types in packages javax, org.xml.sax, etc. The OSGi container will add an implicit Import-Package for all of the java.* packages provided by the JRE. For JRE classes that are not under java.*, the OSGi container will provide these for bundles that request them, but they are not automatically available. If your bundle depends on any JRE type that is not under java.*, you must explicitly declare this in your MANIFEST.MF using the Import-Package directive. For example:

  Import-Package: javax.servlet,
   javax.servlet.http
  

• Rollout Globally and Activate Bots - bots activity level stays at zero. When using the management console and selecting the option to "Rollout Globally and Activate Bots" sometimes the bot activity level remains at zero. If this happens, the desired bot activity level and profile can be set by using the Bots view.

5.1.2   Running the Application in the Clouds View

• Error message: Cannot locate JRE definition: "". Launch aborted. when launching simulator One of the OS X Java updates caused JRE configurations to not propagate correctly. You will need to go to Eclipse preferences-> Java-> Installed JREs and add a new JRE definition. The default values should be- JRE Home: '/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home' and JRE Name: 'JVM 1.6.0 (MacOS X Default)'.

5.1.3   Workspace Errors

• Error message: Cannot find Object class referenced by required class files

  This appears as an error against the first line of source code if Eclipse is improperly installed. The most common causes are that the application is missing plug-ins (specifically JDT) or there is no compatible Java Development Kit found.

• Error message: Failed to find OSGi service for monterey-mediator-factory

  This appears when starting a mediator and the OSGi service required to instantiate the application's code cannot be found. This OSGi service is by default defined using xml declarative services in the META-INF/OSGI-INF directory. This file must declare a valid OSGi service and the names must match those used in the application descriptor configuration file. You can see which OSGi services are registered on a given node via the OSGi console (port 8023 by default on a network node). The following are some common causes and tips for resolving this:

  • A declarative service file must exist for your MediationSegmentServiceFactory class.
  • The component name in your declarative service file must match the segmentService defined in the application descriptor configuration file.
  • The declarative service file must specify a valid implementation class.
  • The declarative service file must provide the interface com.cloudsoftcorp.monterey.network.api.MediationSegmentServiceFactory
  • The application bundles must be installed on the network nodes (and the bundles must be the correct version).

• Error message: Failed to find OSGi service for monterey-lpp-factory

  This is a similar error to the previous one for monterey-mediator-factory. It occurs when the OSGi service for the instantiating the application's LPP code cannot be found. See the suggested steps above, except that the provided interface should be com.cloudsoftcorp.monterey.network.api.LppClientGatewayFactory, and the OSGi service component name should match the clientGateway value in the application descriptor configuration file.

• Error message: Command failed: java.net.SocketTimeoutException: Read timed out

  When executing actions through the management console, such as provisioning and rolling out nodes, there is a timeout for waiting for a response from the management node. When this time elapses, the management console will stop waiting for a response and will display this message. The timeout can cause the rest of the action to be aborted (e.g. when using quick-setup to create a new cubic, if it times out provisioning the nodes then those nodes will not be rolled out). However, any parts of the action that started executing on the management node will continue to execute. Consult the logs on the management node for any errors or other explanations why the command might take so long.

5.1.4   Workspace Warnings

• JRE Container or Execution Environment problems

  Depending on your OS and installed version(s) of Java, Eclipse may show warnings about the JRE Container on classpath not being a "perfect match" to the application environment. We are using Java 1.6 (or higher), and while Eclipse can detect that it is a match, it requires you manually to approve a "perfect match" to get rid of these warnings. This can be done by opening Preferences -> Java -> Installed JREs (Preferences can be found under either the Monterey Studio or the Window item in the menu, or by pressing ctrl-comma) and selecting a 1.6-compatible JVM, then by opening the Installed JREs -> Execution Environment node, selecting the JavaSE-1.6 environment and on the right-hand side, choosing the JVM you just selected.

• Warning message: Serializable class does not declare serialVersionUID

  Java recommends that Serializable classes provide this private static field. This is not necessary for Monterey, so you can disable this warning in the Preferences; or you can use the Eclipse Quick-Fix (Ctrl-1) to do create such a field for you.

5.2.1   Querying the Network

http://localhost:8080/root/status/nodes

List all node IDs.

Note: the representation of a node id is not intended for human consumption - its primary purpose is to allow the IDs to be passed in subsequent web calls.

http://localhost:8080/root/status/nodes?type="MR"

List node IDs for nodes of a given type. Valid values are: LPP, MR, M, TP, SATELLITE_LPP, SPARE and LPP_BACKUP.

http://localhost:8080/root/status/nodes/summary

List a summary of each node.

http://localhost:8080/root/status/nodes/topology

Says which nodes are connected to which. For example, for each LPP it indicates the MR it is connected to.

http://localhost:8080/root/status/segments/summary

List all segments, giving a summary of each.

http://localhost:8080/root/status/segments/allocation

Get the mapping of segment to mediator, indicating where each segment is mastered.

http://localhost:8080/root/status/locationTree

List the locations available for running nodes, and the usage/capacity at each.

http://localhost:8080/root/status/activityModel/machineLoad

For each node, gives the load on that machine (CPU usage and other metrics).

Note: there is an optional 'node' parameter, where the value is in the JSON format returned by /status/nodes. For example, http://localhost:8080/root/status/activityModel/machineLoad?node=...

http://localhost:8080/root/status/activityModel/workrate

For each node, gives metrics about the node's workrate.

5.2.2   Managing Policies

The following requests retrieve information about policies.

http://localhost:8080/root/policy/extantPolicies

Show policies that have been instantiated and not disposed.

http://localhost:8080/root/policy/availablePolicies

Describes the types of policy that are available.

The following requests make modifications, so must be done with HTTP PUT.

http://localhost:8080/root/policy/instantiate?config=...

Instantiates and starts a new policy. The config parameter is a JSON-serialized PolicyConfiguration instance.

http://localhost:8080/root/policy/dispose?id=...

Removes the policy with the given ID, where the ID matches one returned by extantPolicies.

http://localhost:8080/root/policy/suspend?id=...

Pauses the policy with the given ID, where the ID matches one returned by extantPolicies.

http://localhost:8080/root/policy/resume?id=...

Resumes the paused policy with the given ID, where the ID matches one returned by extantPolicies.

5.2.3   Modifying the Network

The following HTTP requests modify the network. They must therefore be done with HTTP POST.

http://localhost:8080/root/controller/rollout?type="MR"&count=1&location=...

Where type can be any of LPP, MR, M, TP, SATELLITE_LPP, and LPP_BACKUP.

The count parameter can be any positive integer.

The location parameter is optional; it should be of the form: "com.cloudsoftcorp.util:com.cloudsoftcorp.util.resource.basic.BasicResourceLocation(\"/London\")"

http://localhost:8080/root/controller/setWorkrateReporting?period=1000

Where period is the duration in milliseconds between reports.

http://localhost:8080/root/controller/provision?count=1&location=...

The count parameter is the number of nodes to provision.

The location parameter indicates where those nodes should be provisioned.

http://localhost:8080/root/controller/migrateSegment?segment="ibm"&node=...

The node parameter identifies the new mediator node.

http://localhost:8080/root/controller/switchoverRouter?node=...&router=...

The node parameter is the source-node, which is to be rewired to the given router.

http://localhost:8080/root/controller/offload?node=...

Moves all work from the given node to other nodes, where possible. For example, a mediator will migrate its segments elsewhere, an MR will reroute any LPP to other MRs, etc.

http://localhost:8080/root/controller/revert?node=...

Reverts the give node so that it becomes "spare".

http://localhost:8080/root/controller/release?node=...

Shuts down the given node.

http://localhost:8080/root/controller/addSegments?segments=...

Creates the given segments. The segments parameter is a JSON collection of SegmentSummary instances.

5.4.1   User Access and SSH

• /root/.ssh/ to contain common private key and authorized_keys for frequent admin users and others
• useful sshd options include:
  • UseDNS no
  • GSSAPIAuthentication no
  • CheckHostIP no
  • StrictHostKeyChecking no (this one also for ssh_config)
• a non-privileged user called monterey also with .ssh set up

5.4.2   Other Packages

• Java 1.6; with certain linux distributions, it will be necessary to enable a "non-free" or "unstable" repository to be able to install the Sun Java packages.
• ant (1.7)
• rsync
• lynx and wget
• nuttcp, netperf and netserver can aid in benchmarking network throughput.
• bc is a command-line mathematical calculator, and can be useful in diagnosing core contention problems between VMs.
• netfront is required if using Solarflare acceleration.

5.4.3   Network Configuration

Unnecessary services should be disabled (daemons and ports); iptables is configured to allow access to ports on the subnet (or disable iptables altogether, especially if you're behind a firewall and want lower latency).

5.4.4   Virtualization Notes

We recommend XenServer 5.5 for the hosts; as there is no requirement for XenServer Essentials, the freely-available version is sufficient.

It is recommended to allocate a core to each VM on the hosts. Create the many identical VMs by from a 'master' VM (see above) that is exported, copied to all hosts, and re-imported as many times as necessary on each.

The master VM was created from one of the standard Xen templates (we primarily use 32-bit Debian Etch but have also had success with 64-bit Centos 5.3).

5.5.1   Inspecting and Modifying the Network

UserPolicy implementations interact with the network through a separate UserPolicyHelper instance, which is typically provided at construction time (by the factory) and cached for later use. This object is responsible for ensuring that network mutations are properly synchronized and isolated from one another, by permitting changes only within the scope of a NetworkAction class. The flow of control between the UserPolicy and NetworkAction proceeds as follows:

• The policy initiates a network change by calling userPolicyHelper.runAction, passing in an instance of NetworkAction as a parameter.
• The NetworkAction is executed automatically via its run method. It is passed a NetworkHandle instance, which exposes many methods for manipulating the network.
• The NetworkAction makes the necessary modifications to the network using the provided NetworkHandle instance.

An illustrative, if not very useful, network action is shown below, which migrates each of the network's active segments to a different (randomly-chosen) mediator:

NetworkAction randomMigrationAction = new NetworkAction() {

    public void run(NetworkHandle handle) {
        Collection<NodeId> mediators = new ArrayList<NodeId>();
        mediators.addAll(helper.getNetworkInfo().getAllMs());

        for (String segment : helper.getNetworkInfo().getAllSegments()) {
            NodeId currentMediator = helper.getNetworkInfo().getSegmentAllocation(segment);
            mediators.remove(currentMediator);
            NodeId targetMediator = CollectionsUtils.getRandomElement(mediators);
            mediators.add(currentMediator);

            handle.migrateSegment(segment, targetMediator);
        }
    }
};

helper.runAction(randomMigrationAction);

5.5.2   Using Custom Policies

To use a custom policy, you must do three things:

1. Package it in an OSGi bundle.
2. Add it to the Monterey application descriptor.
3. Deploy the bundle, along with your application.

<?xml version="1.0" encoding="UTF-8"?>
<scr:component xmlns:scr="http://www.osgi.org/xmlns/scr/v1.1.0" name="my policy">
  <service>
    <provide interface="com.cloudsoftcorp.monterey.network.control.userpolicy.api.UserPolicyFactory"/>
    <provide interface="com.example.MyPolicyFactory"/>
  </service>
  <implementation class="com.example.MyPolicyFactory"/>
</scr:component>

Service-Component: OSGI-INF/my-policy.xml

bundles {
    management {
        activate "com.example.monterey.policy"
    }
}

5.6.1   Application Development

Applications to run on Monterey Middleware must be packaged as OSGi bundles.

An application must define and register OSGi services that can create instances of LppClientGateway and MediationSegmentService for use on LPP and mediator nodes. We recommend that declarative services are used for registering these services.

The first service must implement the interface com.cloudsoftcorp.monterey.network.api.LppClientGatewayFactory, and must be registered with that interface name. It must also define a component name that can be used in the application deployment descriptor file. Similarly for the second service, it must implement the interface com.cloudsoftcorp.monterey.network.api.MediationSegmentServiceFactory and must be registered with that interface name, and it must define a component name. One or more MediationSegmentServiceFactory service can be registered. It is the job of the application deployment descriptor author to choose which service to use for which segments.

Here is an example taken from the chatroom tutorial. First, a MediationSegmentServiceFactory is defined that can instantiate a MediationSegmentService:

package com.acme.chatroom;

public class ChatroomSegmentService implements MediationSegmentService {

    public static class Factory implements MediationSegmentServiceFactory {
        public MediationSegmentService newSegmentService(String segment) {
            return new ChatroomSegmentService();
        }
        public SegmentStateBackup newSegmentBackup(String segment) {
            return null;
        }
    }

    ...
}

Next, an XML file in META-INF/OSGI-INF/serviceFactory.xml declares this as a service. It gives the service instance the name com.acme.chatroom.monterey.serviceFactory:

<?xml version="1.0" encoding="UTF-8"?>
<component
    name="com.acme.chatroom.monterey.serviceFactory"
    xmlns="http://www.osgi.org/xmlns/scr/v1.1.0">
    <service>
        <provide interface="com.cloudsoftcorp.monterey.network.api.MediationSegmentServiceFactory"/>
    </service>
    <implementation class="com.acme.chatroom.ChatroomSegmentService$Factory"/>
</component>

The META-INF/MANIFEST.MF has a clause that ensures this file will be used to define an OSGi declarative service:

Service-Component: META-INF/OSGI-INF/*.xml

The application deployment descriptor file identifies this named service as the one to use for all segments. This is done with the line:

segmentService "com.acme.chatroom.monterey.serviceFactory"

Assuming that the application's project is called com.acme.chatroom and that this matches the bundle's symbolic name, then the descriptor file also declares this bundle:

bundles {
    application {
        activate "com.acme.chatroom"
    }
}

If the node is rolled out as a mediator, the bundle is started which causes our service to be instantiated and registered. Monterey Middleware can then lookup the service using the component name defined in the deployment descriptor; under-the-covers it uses the standard OSGi service lookup technique:

bundleContext.getAllServiceReferences(
    "com.cloudsoftcorp.monterey.network.api.MediationSegmentServiceFactory",
    "component.name=com.acme.chatroom.monterey.serviceFactory");

When the bundle is reverted or released, the bundle is automatically stopped if it was started during rollout.

5.6.2   Application Deployment

Once the OSGi project has been created, there are two ways to deploy it. The project can either be deployed directly from the clouds view or it can first be built into a jar.

To launch directly from the clouds view, select your plugin projects in addition to your application projects and drag to a cloud environment in the clouds view. The OSGi bundles will then be built automatically and deployed to the Monterey network.

You can also package a project as a jar for later use. To create the jar from Monterey Studio, right click on the project and select export. Open the "Plug-in Development" folder and chose "Deployable plug-ins and fragments". In the resulting wizard, ensure that your project is ticked, along with any additional projects it requires. Provide a destination directory when prompted, and click "Finish". The wizard will generate a jar file per project in a "plugins" sub-directory of your specified location; these jar files are the OSGi bundles. The jar files can then be deployed in the same manner as before - dragging to a cloud environment.

Alternatively, bundles can be installed on the network nodes manually. Simply copy them into the plugins directory of the monterey-network-node installation. When the network node is started, the bundles will be installed.

5.9.1   User Interface

The application contributes a custom view to the Eclipse workbench, which shows typical trading data about each of the instruments in the exchange, as well as their current order books of unsatisfied "buy" and "sell" orders.

Selecting a stock from the table on the left will cause its order book to be displayed in the right-hand pane. The order book updates in real-time as orders are placed by connected users (or by programmatic bots for testing or simulation).

5.9.2   Connecting to the Running Application

Each of the EZ-Brokerage client gateways (hosted at the Monterey LPP nodes) starts an embedded web-server, which allows client applications (e.g. the trader view) to retrieve the public state of the order books of all the instruments in the exchange. The client can connect via any of the available LPP nodes; they are all equivalent.

If the view is not connected to any gateway (for instance, if no LPP nodes have been rolled-out in the network, or if the Disconnect action was performed), the view will appear in the disconnected state:

Which client gateway is being used can be specified via the Connect via Gateway menu, accessible from the view menu. If the Automatically Connect option is checked, the view will connect to the first gateway that it discovers at runtime; the connection can be subsequently changed through the Connect via Gateway menu. The Disconnect menu item is disabled whenever this option is selected.