JSON Proxies: Inspecting & Modifying the Payload

How to read and update JSON in OSB. Download the full example.

UPDATE from future! DO NOT CONVERT JSON to XML! It is a pain if you need to convert it back! Use Javascript!
Using Javascript to Inspect & Modify JSON Payload

See other posts about OSB & JSON:
Why JSON Does Help Direct Proxy Performance
How To Build a JSON Pass-Through Proxy in OSB
OSB and JSON Proxies: Gathering Statistics

We can make a pass-through JSON proxy in OSB pretty easily.

Sometimes this is not enough, however, and we need to inspect or even modify the JSON payload.

OSB does not provide us with the tools to do it. So, should we use existing libraries or write our own?

(Existing ones, of course! But careful: see Addendum)

Test Project: Insert User ID if Missing

I’m going to implement a basic OSB project that does JSON inspection and transformation.

I will use a real project as a prototype, though of course for this post I’ve simplified it to get rid of all non-essential details.

The necessary logic goes like this: if the incoming JSON request has a “userid” field with a value, then pass the request downstream as is; otherwise, inject the value “anonymous” into the field.

E.g., here’s a request that should be passed with no modification:

{
  "userid":"john",
  "id":1234
}

While this one

{
  "id":1234
}

will need to be updated to become

{
  "userid":"anonymous",
  "id":1234
}

Parsing Directly into XmlObject for Better Performance

First, we need to get or build Java code to do JSON to XML and XML to JSON transformations.

There are a number of JSON libraries, and some of them have the necessary built-in functionality for converting JSON into some form of XML.

OSB, though, requires XmlBeans’ XmlObject as for representing XML. Using any converter that doesn’t produce XmlObject directly will force us to:

1. Convert JSON into non-XmlBeans XML
2. Serialize that XML into a string
3. Parse that string with XmlBeans

We know that the serialization and parsing of XML are very expensive operations. We need to convert JSON directly into XmlObject if we want a high performance solution.

I failed to find existing XmlBeans code that converts from JSON, despite mentions of JSON in relation to XmlBeans that are scattered all over the Internet.

No problem, all we need is a JSON parser that follows the SAX model, i.e., it generates an event when it parses the next JSON element. We will then generate the corresponding XML element in the XmlObject tree.

From the list of libraries on json.org, I have chosen Argo, which does exactly that with its JsonListener interface.

Java API

I’m not going to describe the details of the Java implementation here. If you’re interested, it is available on a public repository on Bitbucket.

The top-level API though is this:

net.jsontoxmlbeans.Convertor:

public static XmlObject toXML(Object o);
public static String toJSON(XmlObject xml);

The first method can take JSON from a string variable:

let $json = '{"a":123}'
...
Java Callout: toXML($json)

… or from a whole $body where the first child text node contains the JSON:

{"a":123}

...
Java Callout: toXML($body)

… or from any XML variable where the first child text node contains the JSON:

let $json := [1,2,3,4,null] 
...
Java Callout: toXML($json)

The resulting XML will conform to schema. For example, this JSON

[1,"foo",true,false,{"val":123},[5,6,7],null]

will be converted into this XML:

<json xmlns="http://json.to.xml.for.osb">
  <array>
    <number value="1"/>
    <string value="foo"/>
    <boolean value="true"/>
    <boolean value="false"/>
    <object>
      <field name="val">
        <number value="123"/>
      </field>
    </object>
    <array>
      <number value="5"/>
      <number value="6"/>
      <number value="7"/>
    </array>
    <null/>
  </array>
</json>

The toJSON() method takes an XmlObject (expected to be built according to schema) and formats it into a JSON string.

You can download the JAR here.

Test Project Implementation

I use the test project from my previous posts, and update the AddUser operation.

The first thing I do is to import the JAR into the project.

Then, in the entry proxy, I parse the request with toXML($body).

If there is no userid field, then insert it and transform the XML back to JSON.

Easy!

Let’s review the required steps one by one:

Step 1. Execute the Java callout to transform the body content into XML.

Step 2. Check if the userid field is present in JSON.

Step 3. Insert the missing userid field.

Step 4. Execute the Java callout to transform the XML back into JSON and place it into the $json variable.

Step 5. Place JSON into $body.

That’s it! The JSON payload has been inspected and, if required, updated.

BTW, the average time for a two-way transformation of a 200K JSON object in my tests is around 50ms.

Now, on to a horror story:

Addendum: Using the org.json Package (and Failing)

The very first Java library listed on json.org is org.json, and it has a utility class for converting JSON into XML and back. Excellent!

I almost jumped straight to integrating org.json into my test project, but still decided to do some basic testing first. As it turned out, this saved me some embarrassment.

The library’s utility class (named simply XML) takes a parsed JSON object and converts it into XML. The code looks like this:

    JSONObject jo = new JSONObject(json);
    return XML.toString( jo );

At some point during testing, I fed it with this innocent JSON:

    {"bug":"777","bug2":"true"}

and got this XML:

    <bug>777</bug><bug2>true</bug2>

Wait… but where has the type information gone? In JSON, “777” is not a number, but a string. And “true” is not a boolean value, but a string too. The type information is lost!

Needless to say that after converting the result back into JSON with the following code:

    JSONObject jo = XML.toJSONObject(xml);
    return jo.toString();

I got not the original JSON, but a modified (or, rather, corrupted ) version:

    {"bug":777,"bug2":true}

Depending on the target system’s parser, this JSON may or may not be considered valid.

The bigger issue with this convertor is that it may not generate well-formed XML. Consider this perfectly valid JSON:

    {"not well formed":true}

After passing through the org.json convertor it produces this:

    <not well formed>true</not well formed>

This “XML,” of course, won’t be parsed by any compliant XML parser, including Apache XmlBeans used in OSB.

The org.json library is not suitable for use in OSB.