Form Submission with Integration to External Restful API

Introduction

You explore the process of designing a solution for a specific business use case within the Joget platform. Your goal is to ensure that data from a demo request form is successfully sent to an external system, such as CRM software, after the form is validated and processed. You'll learn how to design the solution, configure the form and userview, and explore various methods for integrating with external systems to ensure seamless data transfer.

How does it work?

Problem statement

Let's consider a scenario where a visitor submits a demo request form in your Joget app. Here's how you would handle it:

1. The visitor submits a demo request form using the Joget app.
2. The form fields are validated to ensure all mandatory fields are filled.
3. Data Sharing: Upon successful validation, the form data is shared with an external system, like CRM software, using plugins such as the JSON API Tool or custom Bean Shell code.

The primary objective is to ensure the data is successfully delivered to the external system. The only factor outside of Joget's control might be the integration with the CRM software. This article will guide you through the best design practices for this business use case, keeping UI/UX considerations in mind.

This is an example of what the form would look like.

Designing the solution

You'll start by designing the app and determining the best points to invoke the external API.

Preparing the form and userview

1. Begin by designing the form itself. It should be simple, with just three mandatory fields.
   
2. In the userview, use the Form Menu and link it to the form you've designed.
   
   Configure the Action to Perform After Form Saved to redirect to an HTML Page displaying a submission message (e.g., Form submitted. Thank you!).
3. Do not forget to create a CRUD menu too so that you can browse through all the submissions easily using Generate CRUD.
   

At this point, you haven't yet integrated with the external CRM.

What happens on form submission?

When the end-user clicks the Submit button, the following will take place:

1. Form Validation: Joget checks each form element and invokes the validator if one is configured.
   
   If all validations pass, then it will move to the next step, else, end user will be redirected back to the same form with validation errors displayed like what is shown in the screenshot below.
   

2. Form Store: Once all validations pass, Joget will proceed to the next step; form data will be passed to the Store Form Data.
   
   
   
   
   By default, the load/store form Data is Default where it will load and store form data into the table name declared in the form properties. In this case, the table name is demo_request.
   Since you are using Default, this would also mean that you are saving the form data locally in Joget's database.

With what you have learned so far, this can be presented using the following diagram.

Invoke restful API call

There are several methods available to invoke a RESTful API call:

Method 1 - JSON Tool

The easiest no-code approach is to use the JSON API Tool plugin. This plugin acts as both a Process Tool & Post Form Submission Processing Plugin. It can be invoked either from within a process flow or directly after form submission.

Method 2 - Bean Shell Code

Alternatively, you can write custom Bean Shell code to make HTTP calls. Here's a sample code for making a RESTful API GET call:

import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import java.io.IOException;
import org.joget.commons.util.LogUtil;
 
try{
    String jsonUrl = "http://localhost:8080/jw/web/json/workflow/assignment/list/count?packageId=crm"; //sample url
    String name = "header1";
    String value = "value1";
 
    CloseableHttpClient client = null;
    CloseableHttpClient client = HttpClients.createDefault();
 
    HttpRequestBase request = null;
    request = new HttpGet(jsonUrl);
    request.setHeader(name, value);
 
    HttpResponse response = client.execute(request);
 
} catch (Exception ex) {
    LogUtil.error(getClass().getName(), ex, "");
} finally {
    try {
        if (request != null) {
            request.releaseConnection();
        }
        if (client != null) {
            client.close();
        }
    } catch (IOException ex) {
        LogUtil.error(getClass().getName(), ex, "");
    }
}

This code can be executed from various plugin types, offering flexibility on when and where to invoke it. However, the disadvantage compared to the former is that we need to maintain the custom coding ourselves instead of configuring through a plugin. 

These are the plugin types relevant to our solution to call the code from:

1. Bean Shell for Process Tool
2. BeanShell Validator
3. Bean Shell Form Data Store 

Method 3 - JSON tool from Bean Shell code

You can only combine both methods together by triggering the JSON process Tool plugin in a beanshell code too. Here's a quick sample code written to be used in Bean Shell for Process Tool. Note that each plugin type would inject different variables for the Bean Shell code to consume. 

import java.util.Map;
import java.util.HashMap;
import javax.servlet.http.HttpServletRequest;
import org.joget.apps.app.model.AppDefinition;
import org.joget.apps.app.service.AppPluginUtil;
import org.joget.apps.app.service.AppUtil;
import org.joget.plugin.base.ApplicationPlugin;
import org.joget.plugin.base.Plugin;
import org.joget.plugin.base.PluginManager;
import org.joget.plugin.property.model.PropertyEditable;
import org.joget.workflow.model.WorkflowAssignment;
   
public Object execute(AppDefinition appDef, HttpServletRequest request, WorkflowAssignment workflowAssignment) {
    String jsonUrl = "http://localhost:8080/jw/web/json/workflow/assignment/list/count?packageId=crm"; //sample url
      
    //Reuse Email Tool to send separated email to a list of users;
    Plugin plugin = pluginManager.getPlugin("org.joget.apps.app.lib.JsonTool");
    ApplicationPlugin jsonTool = (ApplicationPlugin) plugin;
 
    Map propertiesMap = new HashMap();
    propertiesMap.put("pluginManager", pluginManager);
    propertiesMap.put("appDef", appDef);
    propertiesMap.put("request", request);
    propertiesMap.put("workflowAssignment", workflowAssignment);
     
    //configure json tool plugin
    propertiesMap.put("jsonUrl", jsonUrl);
    propertiesMap.put("requestType", ""); //empty is for GET call
    propertiesMap.put("multirowBaseObject", "");
    propertiesMap.put("debugMode", "");
    propertiesMap.put("formDefId", "request");
    propertiesMap.put("headers", new Object[]{});
 
    List fieldMappings = new ArrayList();
 
    Map fieldMapping = new HashMap();
    fieldMapping.put("jsonObjectName", "total");
    fieldMapping.put("field", "day");
    fieldMappings.add(fieldMapping);
 
    //repeat this code to add more row
    // fieldMapping = new HashMap();
    // fieldMapping.put("jsonObjectName", "jsonAttrName");
    // fieldMapping.put("field", "formFieldId");
    // fieldMappings.add(fieldMapping);
 
    propertiesMap.put("fieldMapping", fieldMappings.toArray());
          
    //set properties and execute the tool
    ((PropertyEditable) jsonTool).setProperties(propertiesMap);
    jsonTool.execute(propertiesMap);
      
    return null;
}
   
//call execute method with injected variable
return execute(appDef, request, workflowAssignment);

Possible integration Points to Invoke Restful API Call

Method 1 - Post form submission processing and JSON tool

Using Post Form Submission Processing in Form and the JSON API Tool is the quickest method. It allows you to invoke any Process Tool & Post Form Submission Processing Plugin.  JSON API Tool is one such candidate.

• Upon form submission, form fields will be validated, and the form data will be stored. Then, the Post-Form Submission Processing will be triggered.
• The response time for the form submission will now include the complete execution of the JSON Tool.
• If the external JSON API takes longer than expected to respond, the end user will be kept waiting.

• Depending on the feature of the API call, we would assume that it would return a response to indicate successful execution.
  For example:

  { "success" : "true" }

• By using this integration point, there's no way to redirect the user to other place/menu when error occurs.

This is to sum things up so far.

• Pros:

1. 1. Quickest
   2. Easiest to Configure

• Cons:

1. 1. Increased wait time due to JSON Tool's execution time.
   2. No guarantee of successful JSON call to the external system.

Method 2 - Post form submission processing and JSON tool with multi tools

By using Multi Tools, you can avoid waiting for the JSON Tool to complete.

Set the Run Mode to execute the process tool in a new thread.

• Quickest
• Easiest to Configure

Cons:

• No guarantee of successful JSON call to the external system.

Method 3 - Workflow process with JSON tool

With what you have learned so far, there's still 1 con that we are trying to solve. Let's try to put the form within a process flow in a diagram as below.

Using a process flow, you can check the content of the returned JSON call to see if it matches the intended content. For example, you expect this reply and have it mapped to a workflow variable.

{ "success" : "true" }

By doing so, you could redirect back to the same form and prompt the end user to submit the form again. However, if the external API is unreliable, the potential customer (end-user) might be unable to complete the submission process and could repeatedly see the same form. This would result in a confusing UI presentation and a frustrating user experience.

Since this design expects a "true/false" response from the JSON Tool, it might not return anything when the external API service is unresponsive, forcing you to wait for it to time out (around 60 seconds).

Pros:

1. Handles JSON call errors and prompts the user for re-submission

Cons:

1. Increased wait time due to JSON Tool's execution time.
2. Form submission is only end-to-end when the external API service is working.

Method 4 - Form validator and form data store

Triggering the JSON call from these 2 plugin types is not feasible.

As the JSON Tool plugin is not a form validator nor a Form Data Store type of plugin, we will need to use the workaround shared earlier, which is writing the bean shell code from scratch or calling the JSON Tool plugin programmatically.

Form Validator

Form Validator Plugin expects a true/false to be returned. We may try to avoid the need to design a process flow and achieve the same JSON result checking by making the JSON call through the validator plugin. The problem that would arise from this approach are:

1. The same form content may be submitted repetitively, and other form fields' validators may kick in, too (i.e., mandatory field checking). This results in many JSON calls, which is, in most cases, not the desired behavior.
2. If the external API system becomes unresponsive, the Joget app will stop working.
3. Increased wait time. Response time includes JSON Tool's turnaround time.

Form Data Store 

To avoid calling the API repetitively, let's move down to the next layer, Data Store.

By calling the JSON API within the Form Data Store Plugin, we must explore how to handle events, such as when JSON API is not responsive. This type of plugin won't expect a true/false to be returned like the validator plugin.

You can try throwing an exception instead in the Bean Shell code you are writing.

This approach suffers from the following issues:

1. If the external API system becomes unresponsive, the Joget app will stop working. If the intent is not to let the user proceed until the JSON call is successful, then this is a viable choice.
2. The error message is not friendly. This is because the code written is wrapped within the Bean Shell plugin itself, and the exception message from the code does not propagate to the form UI. To mitigate/improve this, we must develop a custom Data Store itself.
3. Increased wait time. Response time includes JSON Tool's turnaround time.

Method 5 - Post form submission processing and JSON tool with multi tools and follow-up on failed API calls

How can you ensure that form submissions are captured without requiring the potential customer to wait for the JSON call to be completed and to continue receiving form submissions even when the external API is unreachable at the time of submission?

Instead of reinventing the wheel, you can extend the capabilities of the current JSON Tool to capture the response's status code and store it in a workflow variable and/or form data. The plugin is in enhanced-json-tool.

Building on Method 2 mentioned earlier, you would place the new plugin in Multi Tools and set the Run Mode to Run tools sequentially in a new single thread. This allows the customer to submit the form without waiting for the JSON call to complete.

The following is a new section you'll configure to capture the JSON call's response status.

You must create a new form to capture the JSON call log.

In the screenshot below, you can inspect each form submission made (on the left) and the result of the API call (on the right).

The highlighted row shows that the API call failed with a response status code of 524.

In the screenshot below, 2 log records are created. The JSON call is successful, but the data response triggered a casting exception.

With the log data on hand, you can create a scheduled task that picks up unsuccessful API calls and attempts to trigger them again later using the following SQL (MySQL).

SELECT req.*, log.c_status as `latest_status` FROM app_fd_demo_request req LEFT JOIN (SELECT MAX(dateCreated) as dateCreated, c_request_id FROM app_fd_demo_request_log log GROUP BY c_request_id) a ON req.id = a.c_request_id JOIN app_fd_demo_request_log log ON log.dateCreated = a.dateCreated WHERE log.c_status != '200' ORDER BY req.dateCreated DESC

The list would look like the screenshot below. The latest log will be displayed for each request row submitted.

Since we're storing the exact form data in Joget's database, you can attempt to make the same JSON call again later.

For example, you can map the Form Update Process Tool Datalist Action to the JSON Tool.

Once you've tested and confirmed that the process works, you can consider automating it. Set up a scheduled job to iterate through the list and execute the JSON Tool using the Iterator Process Tool.