Peer reviewing a NiFi flow

We’re used to doing reviews for source code. NiFi flows look different, but when they are part of your application’s code, it’s useful to have a review prior to committing. Here’s what I look for when I’m reviewing a flow.

Setting an attribute with a dynamic property name

The NiFi EvaluateJsonPath processor lets you specify a JSON path, which will be applied to the content of a flow file. For most situations that works great, but what if you want to retrieve a value where the property you want to read depends on a flow file attribute? EvaluateJsonPath doesn’t support the expression language on dynamic attributes. Use an ExecuteScript processor instead.

Processing upstream deletes

We can load data into a data hub from a variety of upstream data sources. What if we want to delete documents that have been deleted upstream? In the ideal case we’re using a Change Data Capture system and we get notifications about those deletions, but sometimes we don’t have that. How can we detect what’s been deleted if we don’t get notified?

Working with JSON in XQuery

MarkLogic supports XQuery and JavaScript as native languages. XQuery is a very natural way to work with XML, just as JavaScript is a very natural way to work with JSON. However, sometimes it’s useful to cross over and work with JSON using XQuery or vice versa. This post has some tips on using XQuery to work with JSON in memory.