Migrating a Single Joget App

App considerations

App form data

This section applies to you if you intend to migrate the form data stored in the server; otherwise, you can move on to the next section.

As part of application-building best practices, your app table name should have its own prefix.

Example:
Below, the Expenses Claim app uses j_expense as the prefix for the names of its forms' table naming.

Migrating the app form data out would be quite straightforward since it resides in its own set of tables. Depending on the database systems that you are using, you can make use the appropriate tools/clients to export the data out. 

When exporting an app, you can also export its dependent plugins and form data. By default, only the app design is exported.

However, it is important to note that although there's an ability to export form data in the Export App feature, the intent of this feature is to export/import setup data, not production data. If the resultant export package size file size is too big, importing it later on will result in error/request time out issues.

Shared app form data

This section applies to you if you intend to migrate the form data stored in the server; otherwise, you can move on to the next section.

Referring to the example de app form data, you can use the Form builder > Advanced Tools > Table > Table Usage In Other Apps to check if tables are shared with other apps.

If there are shared form data tables, you must assess how to separate the data or continue to access the shared data upon separation. There are a few things to consider based on the assumption that the data is vital to the app that you are migrating out:-

1. How are you using the shared tables? As a lookup, or do you write into them too?
2. Can you continue to provide access from the original server?
3. At which layer should the access be created? Joget app server or DB server level?
4. If DB level, you can consider using JDBC-related plugins(i.e. Database SQL Query Options/ Database SQL Query/ Database SQL Query List Data Store).
5. If Joget app level, you can consider using JSON API or provision new ones using API Builder where you can exert more controls in API management.

App process data

This section applies to you if your app uses processes (Process Builder); otherwise, you can move on to the next section.

Due to the complexity and structure of the data of the workflow engine (shk* tables), you do not recommend migrating process data out in parts.

This is because data of process instances are stored in set of tables shared by all other apps in the same Joget server. Process data contains not just instances from the latest version of the app that you are migrating out; it may also contain all the process instance data of the same app but also from earlier versions. The current server and target server may also contain different sets of versions of the same app. Therefore, migrating the data out in parts will likely affect the normal functionality of the workflow engine.

It is recommended that process instances' data be left behind. This is the appropriate approach for process instances that are still in transit (open.running).

1. Migrate the form data to the new server.
2. Compile a list of process instances that are still running. It should also contain the pending activity and the latest set of workflow variable data.
3. Start a new process instance with reference to the form record ID and set of workflow variable data and resumes at the last pending activity.

This way, end users should be able to continue with their process flow seamlessly.

In case you find it important to preserve the audit trail of the part / completed process data, you can enable Process Data Collector at the app level. Process related data of the app would be written into set of tables (app_report_* tables) that can be easily digested for analytical use such as by using SLA Report Menu. However, this plugin would only work on new process data as they are being executed after it is enabled, and not retrospectively on past process data.

Plugins

This section applies to you if your app is making use of plugins. Otherwise, you can move on to the next section.

Referring to Figure 1 above, plugins that are used by the app can be exported alongside with the app itself. You should check if the new / target server has the same plugins but of different versions and check for its compatibilty.

Integration with external systems

This section is applicable to you if your app is making external calls / relying on external systems, otherwise, you can move on to the next section.

Make a list of all external touch points that your app is using.

The server calls in app's performance tab is a good place to refresh your memory as it will be able to help to list down calls being made outside of the Joget server over a period of time.

For each of them, do check for permission in the external systems to see if you need to whitelist / permit your new server in order to continue to function normally. If the new server is located in environment that is significantly different with current server (i.e geographically far away), it would be good to test its network / throughput to ensure that the app in the new server would be able to function within expected response times and not get bogged down by slow external integrations.

Platform considerations

Joget server version

When you are migrating the app from one to another, it is good to maintain the exact same version from the original server and the one you are migrating to. This is so that we do not introduce too much of moving variables, in case the app we migrated to, fails to function as expected.

Bear in mind that if you intend to upgrade your Joget platform, do it first in the environment where the staging server is or upgrade the staging server where the said apps have been confirmed working before by following the Upgrading Guide For DX 8. This way, the only change made is to upgrade the Joget platform version. It would be easier to find out the cause of any problems as there are not many variables changed.

User management

Typically, service will continue to be provided to the same users on the new server as the scale is expanded.

If you are using an identity management system (i.e. LDAP), then it should be relatively easy to simply point to the same server.

Otherwise, you may need to duplicate the existing user directory date (tables with prefix of dir_ in the database) to the new server.

Migration of running process instances

Before we begin

Before attempting anything, always perform a backup in any (app or database) servers that you will be working on. This is so that there's always something to fallback on, just in case!

Here's the checklist that we would recommend before the migration.

1. The target server has the same app, same process(es), and have been tested to be working end to end (with all integrations, and plugins fitted in). This is so that discrepency in app/process design can be ruled out in case of any problem arised later on.
2. The target server has the same user management, so that the same user assigned, in current server, can be found in target server. This is so that problem that may arise when assigning an user can be ruled out.

Performing the migration

Copy form data

Migrate non-process data over first by copying them over. Remember to copy and not to move, so that the original server remains fully functional and intact.

1. Make a list of process instances that are still running.
2. Copy these data over to the target server.
   1. Form Data from the database
   2. File Upload from wflow/app_formuploads
3. Now that form data and its uploaded files are available in the target server, we can migrate process instances over to the target server.

To verify the data that you have copied them over, make use of CRUD Menu or List Menu in the app to browse through the data imported in.

Migrate process instances

Then, perform the following steps, starting with just 1 process instance. The steps below can be performed using the utility app available for download in this article.

1. Based on the list compiled in step 1, start new process instance in the new server using the same user with reference to the form record ID. Here's a sample of a process instance that is still running.
   
   
   In the Figure above, there are 3 vital information that you need. Use them to call the JSON API in the target server by using JSON API#web/json/workflow/process/start/(*:processDefId). We will also need to copy any workflow variable data over to the new instance too. You can pass on var_* parameters to set the workflow variables' values accordingly.
   
   At this point of time, the new process instance would be waiting at its very first activity.
   
   
   IMPORTANT:
   If you have tools that run right at the start of the process flow, this may cause manipulation of form data state. Take note of potential changes that may occur and devise a way to preserve original data/state.
2. Once a new process instance is started, note down the process instance ID generated, and we will then need to make use of JSON API#web/json/monitoring/activity/start/(*:processId)/(*:activityDefId) to resume the activity where it supposed to be, in the original server.
   
   
   
   Depending on how many pending activities you may have in the specific process instance, you may need to make multiple calls to this JSON API. Note that in the first of such calls, you must set the abortCurrent parameter value to true so that it aborts the first default pending activity instance. Subsequently, set abortCurrent parameter to false.
   
   
   IMPORTANT:
   The assignee of the resumed activity instance may not be correct if its participant mapping relies on performer of past activities. This is because past activities' data is NOT available in the target server.
3. In the target server, attempt to continue with the process assignment, as usual, through List Inbox Menu or Inbox Menu in your app, ideally, to the very end, to validate that it is working as expected.

Once we have validated and refined the steps to migrate a process instance, we can then move on to migrate more process instances over.

Utility app for migration of running process instances

It has created a utility app to perform the 3 steps listed above. It has been tested using MySQL database and on Joget DX 7.

Refer to the diagram below to understand how to use it.

1. Install this app in the current server. Publish the app.
2. Go to setup menu to key in parameters required of the target server.
   
3. At the target server, login as admin, navigate to System Settings > General Settings > API Domain / IP Whitelist and key in the domain / IP of the current / old server. This is so that the target server can accept JSON API calls from current server. We have set up the app.
4. Go to List of Running Process to see the list of running process instances. Select and click Migrate to start migration.
   
   IMPORTANT: 
   If your app has User Notification Plugin or similar notification plugin enabled or another process tool that will immediately run after the process starts, it may be a good idea not to have them enabled while migration is taking place.
5. Observe current server log files and target server's Running Processes to verify its execution status / result.
   An entry will be added to Logs menu if it is successful.
   
   In the List of Running Process menu, the status will change to Migrated with process instance ID furnished based on the data from logs.
   

Download sample app