Month: May 2015

Custom query DSL for NoSQL

Posted on Updated on

Problem:  

Need to support the custom query DSL for Domain objects, so that user can create various queries in SQL like format for NoSQL database.

e.g. Select * from User where firstName = “vinayak” and age > 18 order by id limit 10.

The result will be the same irrespective of underlying storage either Elastic Search, MongoDB or MySql.

Solution:

The solution to this problem is to create custom query language by providing SQL like syntax or grammar. The  query defined using this grammar will be translated to the provided data source query language like elastic search query dsl. The tool evaluated for this purpose is ANTLR which is widely used for designing the custom domain specific languages. I highly appreciate the work done by Terence Parr in this area.

The language translation for target data source using ANTLR works in three different Steps.

Step I:

ANTLR provides java based utility which parse the grammar and generate the code as per the target run time like c#, java, javascript etc… The grammar format is supported in Backus Normal Form

Step II:

The code generated from Step I as per target run time will be used to parse the custom query designed using the respective grammar. The output of Step II is parse tree or Abstract Syntax Tree.

Step III:

To translate the parse tree into desired output or targeted data source query ANTLR also provided the parse tree traversal using Depth First Search. The Step I by default generate the  listener interface and empty base implementation for this interface. The Step I also optionally generate the visitor for the parse tree. The listener methods will be called during parse tree traversal, in this callback methods one can perform respective actions as per the need.

This tool will help you to design your own custom language and its translation as per your need in very simple way. For more information one can explore the documentation and The Definitive ANTLR 4 Reference book.

References:

1. http://www.antlr.org/
2. https://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference
3. http://java.dzone.com/articles/creating-external-dsls-using
4. https://theantlrguy.atlassian.net/wiki/display/ANTLR4/Getting+Started+with+ANTLR+v4

Distributed Transaction with Nosql using 2 Phase Commit

Posted on Updated on

Problem:

Distributed transaction need to be supported across MongoDB and Elastic Search from an application. Inherently MongoDB and Elastic Search is not supporting distributed transactions. Here I am going to explain you how it has been solved using well known 2 Phase commit protocol.

Solution:

Basic components/actors involve in 2 phase commit protocol for MongoDB and elastic search is as shown below:

BDAT

Resource manager for MongoDB and ElasticSearch has registered with Transaction Coordinator to participate in two phase commit. During phase I transaction coordinator will send prepare notification to registered resource manager. Each resource manager can give the response as ready to commit or rollback. During first phase transaction log will be created into Redis and in case of update or delete operation need to perform, the initial state of data is also logged and lock is applied on transnational entity in Redis only. In phase I the actual changes will be done on the resource.

If all participants resource manager voted success in phase I then transaction coordinator will send the commit notification in phase II to all participant.If any one failed in phase I will send the rollback notification to remaining participated resource manager. In Phase II participated resource manager will update the transaction log and release the lock on transaction entity.

If in phase II any participant failed to commit or rollback,then that transaction will be logged as in doubt transaction and eventually watch dog will rollback corresponding changes from participated resource manager.

If transaction coordinator it self failed after or in between phase I or in between phase II, then the watch dog will rollback the transaction after transaction timeout and release the lock. In this case data will be inconsistent state still watch dog perform recovery.

2 phase commit failure scenario and its remedy

1 MongDB fail in phase I Data is consistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Failed & rollback I
t2 dt1 ES Ready To Commit I
t1 dt1 Mongo Rollback not called II
t2 dt1 ES Rollback II
2 ES fail in phase I Data is consistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Ready To Commit I
t2 dt1 ES Failed & rollback I
t1 dt1 Mongo Rollback II
t2 dt1 ES Rollback not called II
3 MongDB fail in phase II commit Data is inconsistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Ready To Commit I
t2 dt1 ES Ready To Commit I
t1 dt1 Mongo Commit Failed II
t2 dt1 ES Commited II
t3 dt1 TC InDoubt End of phase II
Wacthdog Action Retry to rollback transaction and release lock
4 ES fail in phase II commit Data is inconsistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Ready To Commit I
t2 dt1 ES Ready To Commit I
t1 dt1 Mongo Commited II
t2 dt1 ES Commit Failed II
t3 dt1 TC InDoubt End of phase II
Wacthdog Action Retry to rollback transaction and release lock
5 MongDB fail in phase II rollback Data is inconsistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Ready To Commit I
t2 dt1 ES Ready To Commit I
t1 dt1 Mongo rollback Failed II
t2 dt1 ES rollback II
t3 dt1 TC InDoubt End of phase II
Wacthdog Action Retry to rollback transaction and release lock
6 ES fail in phase II rollback Data is inconsistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Ready To Commit I
t2 dt1 ES Ready To Commit I
t1 dt1 Mongo rollback II
t2 dt1 ES rollback Failed II
t3 dt1 TC InDoubt End of phase II
Wacthdog Action Retry to rollback transaction and release lock
7 Transaction Coordinator failed before phase I Data is consistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Ready 0
t2 dt1 ES Ready 0
t1 dt1 Mongo waiting I
t2 dt1 ES waiting I
Wacthdog Action No action needed just delete the transaction
8 Transaction Coordinator failed after phase I Data is inconsistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Ready To Commit I
t2 dt1 ES Ready To Commit I
t1 dt1 Mongo waiting II
t2 dt1 ES waiting II
Wacthdog Action Retry to rollback transaction and release lock
9 Transaction Coordinator failed inbetween phase II Data is inconsistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Ready To Commit I
t2 dt1 ES Ready To Commit I
t1 dt1 Mongo Commited II
t2 dt1 ES waiting II
Wacthdog Action Retry to rollback transaction and release lock
10 Transaction Coordinator failed after phase I Data is inconsistent
TransactionLog ID Distributed Transaction Id Participant State Phase
t1 dt1 Mongo Failed & rollback I
t2 dt1 ES Ready To Commit I
t1 dt1 Mongo waiting II
t2 dt1 ES waiting II
Wacthdog Action Retry to rollback transaction and release lock

2 Phase commit is supported by .Net framework since 2.0 using System.Transaction package. It has been extensible for any other resource like MongoDB, Sql, Message Queue … etc using the IEnlistmentNotification interface.

The source code for POC is added here