WSO2 EI is a configuration driven integration software that is best in its category.
WSO2 EI started as WSO2 ESB from a time where SOAP services dominated the internet. Now, more than ten years later, we are in a REST dominated world where JSON is the most used data exchange format.
Thought ESB is evolved to EI, the core still uses XML for data representation and processing. This means even when we are working with a JSON payload, it converted to XML before processing.
This back and forth conversion is introducing several problems as follows.
- Losing namespace information when converted to JSON
- Losing array information when converted to XML ( single element array converted to an object )
- Losing data type information when converted to JSON. ( since XML does not have data types numerical, boolean information will be lost )
- Problems in handling null values.
With the adoption of JSON, ESB was equipped with some configurations to fix these problems.
But, these properties do not allow any fine-tuned changes. For example, let’s consider the auto-primitive property which enables us to automatically detect and parse values to native data-types.
- We might need auto-primitive to be enabled for a proxy and disabled for another. Which is not possible since this is a globally applied property.
- There can be instances that we need auto-primitive detection for selected keys, which is also not possible . Ex : { “person” : { “id” : “001”, “age” : 28 } } == <person><id>001</id><age>28</age></person> we need age to be converted to int, but not the id.
Native JSON support
From EI 6.5.0 onwards, it contains the native JSON support for all possible mediators. ( for example, certain mediators like XSLT mediator are completely XML driven and cannot introduce JSON support )
- Aggregate mediator
- Enrich mediator
- For each mediator
- Iterate mediator
Since these mediators process the JSON input as it is, it’s free from issues happening in the JSON -> XML -> JSON conversions. So, so long as the input JSON payload is going through native JSON supported mediators and content-unaware mediators we will not face any problem.
But, if the JSON -> XML conversion is mandatory and its a part of our mediation flow, ( EX: we have an XSLT mediator in the middle ) the issue still remains.
JSON Transform Mediator
The JSON Transform mediator is the newest addition to the mediator collection in EI. Which is introduced to resolve the above described two issues.
<jsontransform [schema="string"]> <property name="string" value="string"/>* </jsontransform>
JSON Transform mediator is capable of formatting the existing JSON message in the mediation flow, according to a given schema, properties or both. Usually, the JSON Transform mediator is expected to use at the end of mediation flows. ( transform the JSON payload Just before responding or sending to an endpoint )
Sample Usage 01: only schema
<jsontransform schema="conf:/schema.json"/>
Here, we are transforming the current message according to a given schema. For example, we can change the data type of the value from string to int of a given key ( ex; age ), using the schema. ( please refer the example )
Sample Usage 02: only properties
<jsontransform> <property name="synapse.commons.enableXmlNullForEmptyElement" value="true"/> <property name="synapse.commons.json.preserve.namespace" value="true"/> </jsontransform>
Here, we can configure one or more properties with the required values. This enables us to use those properties which earlier had the system level scope, now in the sequence level. Which means we can enable the auto-primitive in one proxy and disable in another.
Sample Usage 03: both schema and properties
<jsontransform schema="conf:/schema.json"> <property name="synapse.commons.enableXmlNullForEmptyElement" value="true"/> <property name="synapse.commons.json.preserve.namespace" value="true"/> </jsontransform>
In this configuration, properties get applied first and schema applied afterward. Suppose we can get the majority of the changes done using properties. For the rest of the fields which need to be changed, we can use the schema. This minimizes the effort required when creating the schema since we need to address a few keys only.
Example
In this example, we are sending a JSON payload to EI and try to get it transformed according to a given schema.
Save the following JSON schema to a file and add it to the registry.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "$schema": "http://json-schema.org/draft-04/schema#", | |
| "type": "object", | |
| "properties": { | |
| "fruit": { | |
| "type": "string", | |
| "minLength": 4, | |
| "maxLength": 6, | |
| "pattern": "^[0-9]{1,45}$" | |
| }, | |
| "price": { | |
| "type": "number", | |
| "minimum": 2, | |
| "maximum": 20, | |
| "exclusiveMaximum": 20, | |
| "multipleOf": 2.5 | |
| }, | |
| "simpleObject": { | |
| "type": "object", | |
| "properties": { | |
| "age": { | |
| "type": "integer" | |
| } | |
| }, | |
| "required": [ | |
| "age" | |
| ] | |
| }, | |
| "simpleArray": { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "boolean" | |
| }, | |
| { | |
| "type": "boolean" | |
| }, | |
| { | |
| "type": "string" | |
| } | |
| ], | |
| "minItems": 1, | |
| "maxItems": 3, | |
| "uniqueItems": false | |
| }, | |
| "objWithArray": { | |
| "type": "object", | |
| "properties": { | |
| "marks": { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "integer" | |
| }, | |
| { | |
| "type": "integer" | |
| }, | |
| { | |
| "type": "integer" | |
| }, | |
| { | |
| "type": "integer" | |
| } | |
| ] | |
| } | |
| }, | |
| "required": [ | |
| "marks" | |
| ] | |
| }, | |
| "arrayOfObjects": { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "object", | |
| "properties": { | |
| "maths": { | |
| "type": "integer" | |
| } | |
| }, | |
| "required": [ | |
| "maths" | |
| ] | |
| }, | |
| { | |
| "type": "object", | |
| "properties": { | |
| "physics": { | |
| "type": "integer" | |
| } | |
| }, | |
| "required": [ | |
| "physics" | |
| ] | |
| }, | |
| { | |
| "type": "object", | |
| "properties": { | |
| "chemistry": { | |
| "type": "integer" | |
| } | |
| }, | |
| "required": [ | |
| "chemistry" | |
| ] | |
| } | |
| ] | |
| }, | |
| "singleObjArray": { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "number" | |
| } | |
| ] | |
| }, | |
| "nestedObject": { | |
| "type": "object", | |
| "properties": { | |
| "Lahiru": { | |
| "type": "object", | |
| "properties": { | |
| "age": { | |
| "type": "integer" | |
| } | |
| }, | |
| "required": [ | |
| "age" | |
| ] | |
| }, | |
| "Nimal": { | |
| "type": "object", | |
| "properties": { | |
| "married": { | |
| "type": "boolean" | |
| } | |
| }, | |
| "required": [ | |
| "married" | |
| ] | |
| }, | |
| "Kamal": { | |
| "type": "object", | |
| "properties": { | |
| "scores": { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "integer" | |
| }, | |
| { | |
| "type": "integer" | |
| }, | |
| { | |
| "type": "integer" | |
| } | |
| ] | |
| } | |
| }, | |
| "required": [ | |
| "scores" | |
| ] | |
| } | |
| }, | |
| "required": [ | |
| "Lahiru", | |
| "Nimal", | |
| "Kamal" | |
| ] | |
| }, | |
| "nestedArray": { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "integer" | |
| }, | |
| { | |
| "type": "integer" | |
| }, | |
| { | |
| "type": "integer" | |
| } | |
| ] | |
| }, | |
| { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "boolean" | |
| }, | |
| { | |
| "type": "boolean" | |
| } | |
| ] | |
| }, | |
| { | |
| "type": "array", | |
| "items": [ | |
| { | |
| "type": "string" | |
| }, | |
| { | |
| "type": "string" | |
| } | |
| ] | |
| } | |
| ] | |
| }, | |
| "allNumericArray": { | |
| "type": "array", | |
| "items": { | |
| "type": "integer" | |
| }, | |
| "minItems": 1, | |
| "maxItems": 3, | |
| "uniqueItems": true | |
| }, | |
| "nullArray": { | |
| "type": "array", | |
| "items": { | |
| "type": "null" | |
| } | |
| } | |
| }, | |
| "required": [ | |
| "fruit", | |
| "price", | |
| "simpleObject", | |
| "simpleArray", | |
| "objWithArray", | |
| "arrayOfObjects", | |
| "singleObjArray", | |
| "nestedObject", | |
| "nestedArray" | |
| ], | |
| "additionalProperties": true, | |
| "minProperties": 10, | |
| "maxProperties": 20, | |
| "patternProperties": { | |
| "_goals$": { | |
| "type": "integer" | |
| } | |
| } | |
| } |
Then add the following proxy service
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <?xml version="1.0" encoding="UTF-8"?> | |
| <proxy xmlns="http://ws.apache.org/ns/synapse" name="testJSONTransform" startOnLoad="true" statistics="disable" trace="disable" transports="http,https"> | |
| <target> | |
| <inSequence> | |
| <jsontransform schema="conf:/schema.json"/> | |
| <respond/> | |
| </inSequence> | |
| </target> | |
| <description/> | |
| </proxy> | |
Finally, test the service using the following CURL
curl -X POST \
http://lahirumadu:8280/services/testJSONTransform \
-H 'cache-control: no-cache' \
-H 'content-type: application/json' \
-H 'postman-token: 38ac4a32-294a-8ae5-4314-a3f3181e693c' \
-d '{
"fruit" : "12345",
"price" : "7.5",
"simpleObject" : {"age":"234"},
"simpleArray" : ["true","false","true"],
"objWithArray" : {"marks":["34","45","56","67"]},
"arrayOfObjects" : [{"maths":"90"},{"physics":"95"},{"chemistry":"65"}],
"singleObjArray" : "1.618",
"nestedObject" : {"Lahiru" :{"age":"27"},"Nimal" :{"married" :"true"}, "Kamal" : {"scores": ["24",45,"67"]}},
"nestedArray" : [[12,"23",34],["true",false],["Linking Park","Coldplay"]],
"allNumericArray" : ["3","1","4"],
"Hello" : 890,
"league_goals" : "10",
"nullArray" : [null,null,null]
}'
Not a silver bullet
In most of the cases, EI is used to content-aware mediation where we know the format of the input payload, passthrough or header-based routing scenarios.
But in a rare requirement, we might not know the format of the incoming XML payload and may need to convert it to JSON and send it to an endpoint.
In such a scenario where we cannot get a schema for the payload, the Transform mediator cannot be used. In fact, such a problem is un-solvable theoretically.
Next steps
At the moment, the Transform mediator
- supports JSON-schema draft 07 and we need to keep supporting the latest releases of the schema.
- expects the user to upload correct schema files and does not validate them prior to the execution. This needs to be fixed.
Thank You 🙂
Lahiru Madushanka 