Data Management Patterns for Microservices Architecture


Data is the primary requirement of any software. Thus, efficient and effective data management can make or break a business. For starters, you have to ensure that data is available to the end user at the right time. Monolithic systems are notorious for their complex handling of data management. In contrast, microservices architecture paints a different picture. Here are some of data management patterns for this type of architecture.

Database Per Service

In this model, data is managed separately by each microservice. This means that one microservice cannot access or use the data of another one directly. In order to exchange data or communicate with each other, a number of well-designed APIs are required by the microservices.

However, the pattern is one of the trickiest to implement. Applications are not always properly demarcated. Microservices require a continuous exchange of data to apply the logic. As a result, spaghetti-like interactions develop with different application services.

The pattern’s success is reliant on carefully specifying the application’s bounded content. While this is easier in newer applications, large systems present a major problem at hand.

Among the challenges of the pattern, one is to implement queries that can result in the exposure of data for various bounded contexts. Other options include the implementation of business transactions that cover multiple microservices.

When this pattern is applied correctly, the notable benefit of it includes loose coupling for microservices. In this way, your application can be saved from the impact-analysis-hell. Moreover, it helps in the individual scaling up of microservices. It is flexible for software architects to select a certain DB solution while working with a specific service.

Shared Database

When the complexity of database per service is too high, then a shared database can be a good option. A shared database is used to resolve similar issues while using a more relaxed approached as a single database receives access by several microservices. Usually, this pattern is considered safe for developers because they can make use of existing techniques. Conversely, doing such always restricts them from using microservices at its best. Software architects from separate teams require cooperation to modify the schema of tables. It is possible that the runtime conflicts occur in case two or more services attempt using a single database resource.

API Composition

In a microservices architecture, while working with the implementation of complex queries, API composition can be one of the best solutions. It helps in the invocation of microservices for the needed arrangement. When results are fetched, a join (in-memory) is executed of the data after which the consumer receives it. The pattern’s drawback is its utilization of in-memory joins—particularly when they are unnecessary—for bigger datasets.

Command Query Responsibility Segregation

Command Query Responsibility Segregation (CQRS) becomes useful while dealing with the API composition’s issues.

In this pattern, the domain events of microservices are ‘listened’ by an application which then updates the query or view database accordingly. Such a database can allow you to handle those aggregation queries which are deemed complex. It is also possible to go with the performance optimization and go for the scaling up of the query microservices.

On the flipside, this pattern is known for adding more complexity. Suddenly it forces that all the events should be managed by the microservice. As a consequence, it is prone to get latency issues as the view DB exercises consistency in the end. The duplication of code increases in this pattern.

Event Sourcing

Event sourcing is used in order to update the DB and publish an event atomically. In this pattern, the entity’s state or the entity’s aggregate in the form of events—where states continue to change—are stored. Insert and update operations cause the generation of a new event. Events are stored in the event store.

This pattern can be used in tandem with the command query responsibility segregation. Such a combination can help you fix issues related to the maintenance of data and event handling. On the other hand, it has a shortcoming as well; the imposition of an unconventional programming style. Moreover, eventually, the data is consistent, not always the best factor for all the applications.

Saga

When business transactions extend over several microservices, then the saga pattern is one of the best data management patterns in a microservices architecture. A saga can be seen as simply local transactions—in a sequence or order. When Saga is used to perform a transaction, an event is published by its service. Consequently, other transactions follow after being invoked due to the prior transaction’s output. In case, failure arises for any of the chain’s transactions, a number of transactions (as compensation) are executed by the Saga to repair the previous transactions’ effect.

In order to see how Saga works, let’s consider an instance. Consider an app which is used for food delivery. If a customer requests for an order of food, then the following steps happen.

  1. The service of ‘orders’ generates an order. In this specific time period, a pending state marks the order. The events chain is managed by a saga.
  2. The service of ‘restaurant’ is contacted by the saga.
  3. The service of ‘restaurant’ begins the process to start the order for the selected eatery. When the eatery confirms, a response is generated and sent back.
  4. The response is received by the Saga. Considering the response contents, it can either proceed with the approval or rejection of the order.
  5. The service of ‘orders’ modifies the order state accordingly. In the scenario of the order approval, the customer receives the relevant details. In the scenario of order rejection, the customer receives the bad news in the form of an apology.

By now, you might have realized that such an approach is too distinct from the point-to-point strategy. While this pattern may add complexity, it is a highly formidable solution to respond to several tricky issues. Though, it is best to use it occasionally.

Advertisement

Are Microservices the Right Fit For You?


The term Microservices was originally coined in 2011. Since then it has been on the radars of modern development organizations.  In the following years, software architecture has gained traction in various IT circles. According to a survey, the enterprises which used microservices were around 36 percent while 26 percent were thinking to include it in the future.

So, why exactly should you use microservices your company? There has to be something unique and more rewarding in it that can compel you to leave your traditional architecture in favor of it. Consider the following reasons to decide for yourself.

Enhance Resilience

Microservices can help to decouple and decentralize your complete application into multiple services. These services are distinct because they operate independently and are separate from each other. As opposed to the conventional monolithic architecture in which code failure can disrupt one function or service, there are little to no possibilities a single service failure to affect another. Moreover, even if you have to do maintain code for multiple systems, it will not be noted by your users.

More Scalability

In a monolithic architecture, when developers have to scale a single function, they have to tweak and adjust other functions as well. Perhaps, one of the biggest advantages of microservices is the scalability which it brings to the table. Since all the services in microservices architecture are separate, therefore it is possible to scale one service or function without having to worry about scaling up the complete application. You can deploy critical business services on different servers to improve the performance and availability of your application whereas your other services remain unaffected.

Right Tool for the Right Task

Microservices ensure that a single vendor does not make you pigeonholed. It can help you to infuse greater flexibility for your projects so rather than trying to make things work with a single tool, you can instead look up for the right tool which can fit your requirements. Each of your services can use any framework, programming language, technology stack, or ancillary services. Despite this heterogeneousness, they can still communicate and connect easily.

Promotion of Services

In microservices, there is no need to rewrite and adjust the complete codebase if you have to change or incorporate a new feature in your application. This is because microservices are ‘loosely coupled’. Therefore, you only have to modify a single service if it is required. The strategy to code your project in smaller increments can help you to test and deploy them independently. In this way, you can promote your services and application quickly, as soon as you complete one service after another.

Maintenance and Debugging

Microservices can help you to test and debug applications easily. The use of smaller modules via continuously testing and delivery means that you can create applications from bugs and errors, thereby improving the reliability and quality of your projects.

Better ROI

With microservices, your resource optimization is instantly improved. They allow different teams to operate by using independent services. As a result, the time needed to deploy is reduced. Moreover, the time for development is also significantly decreased while you can achieve greater reusability as well for your project. The decoupling of services also means that you do not have to spend much on high-priced machines. You can use the standard x86 machines as well. The efficiency which you get from microservices can minimize the costs of infrastructure along with the downtime.

Continuous Delivery

While working with a monolithic architecture, dedicated teams are needed to code discrete modules like front-end, back-end, database, and other parts of the application. On the other hand, microservices allow project managers to add cross-functional teams in the mix who can manage the application lifecycle through a delivery model which is entirely continuous in nature. When testing, operations, and development teams use a single service at the same time, debugging and testing is quickened and made easier. This strategy can help you to develop, test, and deploy your code ‘continuously’. Moreover, you do not have to write new code, instead, you can write code with the help of the existing libraries.

Considerations before Deciding to Use Microservices

If you have decided to use a microservices-based architecture, then review the following considerations.

The State of Your Business

To begin with, you have to think if your business is big enough that it warrants your IT team to work on complex projects independently. If you are not, then it is better to avoid microservices.

Assess the Deployment of Components

Analyze the components and functions of your software. If there are two or more components which you deploy in your project which are completely separate from each other in terms of business processes and capabilities, then it is a wise option to use microservices.

Decide if Your Team Is Skilled for the Project

The use of microservices allows project managers to use smaller teams for development that are well-skilled in their respective expertise. As a result, it helps to quickly generate new functionalities and release it.

Before you adopt the microservices architecture, you have to make sure that your team members are well positioned to operate with continuous integration and deployment. Similarly, you have to see if they can work in a DevOps culture and are experienced enough to work with microservices. In case, they are not good enough yet, you can focus on creating a group who is able to fulfill your requirements to work with microservices architecture. Alternatively, you can also hire experienced individuals to make up a new team.

Define Realistic Roadmap

Exponential scaling is the key to success. Despite the importance of businesses to be agile, it is not necessary for all businesses to scale. If you feel that complexity cannot help you much, then it is better to avoid a microservices architecture. You have to decide on some realistic goals about how your business is going to operate in the future to decide if the adoption of microservices architecture can reap your benefits.

What Is AIOps?


Recently I came across one of the very interesting topics- AIOps.

AIOps refers to Artificial Intelligence for IT operations. It involves the use of machine learning, big data analytics, and AI tools to automate the IT infrastructure of an organization.

In larger enterprises, the applications, systems, and services generate massive data volumes. With AIOps, organizations can utilize this data for monitoring their assets and examining their IT dependence more closely.

Capabilities

Ideally, an AIOps solution provides the following functionalities.

1- Automation for Routine Procedures

AIOps facilitates organizations to integrate automation in daily routine procedures. This automation can be performed for requests from users or to manage non-critical notifications from the system. For instance, if AIOps is used, then a help desk system can respond appropriately to a request from a user—all by itself. Similarly, AIOps tools can assess an alert from a system and evaluate if it requires any action without the need of a supervising authority.

2- Detection

AIOps can detect critical issues quicker and better than any manual strategy. When a familiar malware is detected on a non-critical system, the IT experts may try to eliminate it. In the meantime, they might miss an unusual activity or process on a critical system from a newly-arrived and sophisticated threat. As a consequence, the organization suffers a huge setback.

On the other hand, AIOps can make a difference by the use of vulnerability prioritization .i.e. it immediately notifies the authority about a possible cyber invasion for the critical system while for the non-critical system, it can respond by running an anti-malware tool.

3- Streamlining Interactions

Historically, before AIOps was in the scene, teams had to share and work on information through either meetings or exchanging data manually. AIOps can streamline the communication and coordination processes between teams and data center groups. It shows “relevant” data to all the IT groups. For this, the AIOps platform must be designed in a way so that it can monitor and analyze which type of data to present to which of the team.

Technologies

AIOps combines several techniques for aggregation, algorithms, analytics, data output, visualization, machine learning, and automation and orchestration. All of these techniques are mature and integrated.  So how do they work?

Log files, helpdesk ticketing systems, and monitoring provide data for AIOps. Then big data tools are used to properly manage and aggregate any data coming from the system as an output and convert it into a more useful format. To do this, analytics methods and procedures are used which attempt to extract raw data and transform it into a meaningful fresh piece of data. Therefore, analytics eliminates “noise” and irrelevant data. Additionally, it also searches for recurring patterns that can detect and mark common issues.

However, analytics cannot run without the use of proper algorithms. Algorithms support an AIOps solution to respond with the most appropriate course of action. Algorithms are configured to ensure that the IT staff can help the platform learn about the decisions pertaining to the application performance.

Algorithms are the center of machine learning. The AIOps platform sets out a standard for normal activities and behavior where it can continue to update by adding new algorithms with the addition of new data in the infrastructure of the organization.

Automation ensures that any AIOps tool is quick in performing the required action or set of actions. Automated tools are “forced” to act based on their communication with machine learning and analytics tools. For instance, a machine learning tool may establish that an application in a system requires additional storage to function. This piece of information is passed out to an automation tool which resolves to perform an action like adding more storage.

Lastly, to help in decision making, visualization is used in AIOps. It generates dashboards which are extremely easy to use and read. These dashboards contain graphical representations of all kinds, reports, and other visual elements to simplify different types of output. As a result, the management is able to remain in the loop and take any rational decision.

How Has AIOps Proved to Be a Breakthrough?

Before the emergence of AIOps, organizations faced difficulties because their IT personnel spent much of their time on routine and basic tasks. AIOps proved to be a breakthrough by helping organizations focus on more critical issues. As a result, such platforms have saved a great deal of time. IT personnel now attempt to train and educate AIOps platforms to become familiar with the organization’s IT infrastructure.

Afterward, it continues to update and evolve by making use of machine learning and algorithms as well as going through the “learned” history which it accumulated with the passage of time. Therefore, they provide for an excellent monitoring tool that has the “rationality” to perform many useful tasks.

Moreover, AIOps platforms examine and inspect causal relationships from various services, resources, data sources, and systems. Machine learning functionalities identify and run robust root cause analysis. As a result, troubleshooting of frequent issues is enhanced.

Furthermore, AIOps assists organizations to increase collaboration among all the departments. With the reports from visualization, team leaders are able to comprehend requirements and perform their duties with a renewed sense of direction.

The Other Side of the Coin

AIOps is extremely promising, but some analysts consider it to be unrefined. The debate that the effectiveness of an AIOps platform is as powerful as its “training” while the time needed to create, implement, and administer such a platform may be too time-consuming for many organizations.

Likewise, they argue that due to its ability to perform a wide variety of tasks, it requires trust from organizations. Since AIOps tool works autonomously, they have to be trained in such a way that they can easily adapt according to the environment of their organization and be able to accumulate and collect data, come up to the most logical conclusion, and allocate actions accordingly.

An Introduction to Multikey Indexes with Examples


After understanding single-field and compound indexes, now is the time to learn about multikey indexes. When a field has an array value, an index key is generated for each of its array elements. These indexes are referred as multikey indexes. They can be used with arrays that have scalar values as well as those with nested documents.

To generate a multikey index, you have to use the standard db.collection.createIndex() method. Such indexes are automatically generated by MongoDB whenever it senses an indexed field specified as an array. Hence, there is no need for explicit definition of a multikey index.

Examples

While working with a standard array, let’s suppose we have a collection “student”.

{ _id: 21, name: “Adam”, marks: [ 80, 50, 90 ] }

To build an index on the “marks” field, write the following query.

db.student.createIndex( { marks: 1 } )

As the marks field is an array, thus this index is an example of a multikey index. All of the keys (80, 50, and 90) in its elements point to the same document.

To create a multikey index for array fields having embedded documents, let’s suppose we have a collection “products”.

{

_id: 5,

name: “tshirt”,

details: [

{ size: “large”, type: “polo”, stock:50 },

{ size: “small”, type: “crew neck”, stock:40 },

{ size: “medium”, type: “v neck”, stock: 60 }

]

}

{

_id: 6,

name: “pants”,

details: [

{ size: “large”, type: “cargo”, stock: 35 },

{ size: “small”, type: “jeans”, stock: 65 },

{ size: “medium”, type: “harem”, stock: 10 },

{ size: “large”, type: “cotton”, stock: 10 }

]

}

{

_id: 7,

name: “jacket”,

details: [

{ size: “large”, type: “bomber”, stock: 35 },

{ size: “medium”, type: “leather”, stock: 25 },

{ size: “medium”, type: “parka”, stock: 45 }

]

}

We can build a multikey index with the details.size and details.stock fields.

db.products.createIndex( { “details.size”: 1, “details.stock”: 1 } )

This index is now good to go against queries which have only a single field of “details.size” as well as queries having the both of the indexed fields. As such, these types of queries would benefit from the index.

db.products.find( { “details.size”: “medium” } )

db.products.find( { “details.size”: “small”, “details.stock”: { $gt: 10 } } )

Bounds in Multikey Index

Bounds represent the limits of an index .i.e. how much it needs to scan for searching a query’s results. If there are more than a single predicate with an index, then MongoDB integrates them through compounding or intersection. So what do we mean by intersection and compounding?

Intersection

Bounds intersection point towards the presence of “AND” (logical conjunction) for bounds. For example, if there are two bounds [ 4, Infinity]  and [ – Infinity, 8 ], then intersection bounds process [[4, 8 ]] When $elemMatch operation is used, then MongoDB applies intersection on multikey index bounds.

 

What Is $elemMatch?

Before moving forward, let’s understand the use of $elemMatch first.

$elemMatch is used for matching documents in array field where at the bare minimum, atleast one of the element is matched with the query. Bear in mind, that the operator does not work with $text and $where operators. For a basic example, consider a “student” collection.

{ _id: 8, marks: [ 72, 75, 78 ] }

{ _id: 9, marks: [ 65, 78, 79 ] }

The following query only processes a match with documents in which the “marks” array has at least a single element which is less than 75 and greater or equal to 70.

db.student.find(

{ marks: { $elemMatch: { $gte: 70, $lt: 75 } } }

In response, the result set is comprised of the following output.

{ “_id” : 8, “marks” : [72, 75, 78 ] }

Despite the fact that both 75 and 78 do not conform to the conditions but because 72 had a matched, hence the $elemMatch selected it.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

For instance, we have a collection student which has a field “name” and an array field “marks”.

{ _id: 4, name: “ABC”, marks: [ 1, 10 ] }

{ _id: 5, name: “XYZ”, marks: [ 5, 4 ] }

To build a multikey index with the “marks” array.

db.student.createIndex( { marks: 1 } )

Now, the following query makes use of $elemMatch which means that the array must have at least one element which fulfils the condition of both predicates.

db.student.find( { marks : { $elemMatch: { $gte: 4, $lte: 8 } } } )

Computing the predicates one by one:

  • For the first predicate, the bounds are equal to or greater than 4 [ [ 4, Infinity ] ].
  • For the second predicate, the bounds are equal to or less than 8 [ [ – Infinity, 8 ] ].

Since $elemMatch is used here, therefore MongoDB can apply intersection on the bounds and integrate it like the following.

marks: [ [ 4, 8 ] ]

On the other hand, if the $elemMatch is not used, then MongoDB applies intersection on the multikey index bounds. For instance, check the following query.

db.student.find( { marks : { $gte: 4, $lte: 8 } } )

The query processes the marks array for atleast a single element which is equal to or greater than 4 “AND” atleast a single element which is equal to or less than 8. However, it is not necessary for a single element to conform to the requirements of both predicates, hence MongoDB does not apply intersection on the bounds and uses either [[4, Infinity]] or [[-Infinity, 8]].

Compounding

Compounding bounds means the use of bounds with compound index. For example, if there a compound index { x: 1, y:1 } which has a bound on the x field  [[4, Infinity] ] and a bound on the field y [[-Infinity, 8]]. By applying compounding,

{ x: [ [ 4, Infinity ] ], y: [ [ -Infinity, 8 ] ] }

Sometimes, MongoDB is unable to apply compounding on the given bounds. For such scenarios, it uses the bound on the leading field which in our example is x: [ [4, Infinity] ].

Suppose in our example indexing is applied on multiple fields, where one of the fields is an array. For instance, we have the collection student which stores the “name” and “marks” field.

{ _id: 10, name: “Adam”, marks: [ 1, 10 ] }

{ _id: 11, name: “William”, marks: [ 5, 4 ] }

Build a compound index with the “name” and the “marks” field.

db.student.createIndex( { name: 1, marks: 1 } )

In the following query, there is a condition which applies on both of the indexed keys.

db.student.find( { name: “William”, marks: { $gte: 4 } } )

Computing these predicates step-by-step.

  • For the “name” field, the bounds for the “William” predicate are the following [ [ “William”, “William” ] ].
  • For the “marks” field, the bounds for { $gte: 4 } predicate are [ [ 4, Infinity ] ].

MongoDB can apply compounding on both of these bounds.

{ name: [ [ “William”, “William” ] ], marks: [ [ 4, Infinity ] ] }

 

 

 

 

 

 

 

What Are Indexes in MongoDB?


What Are Indexes in MongoDB?

When a query is run in MongoDB, the program initiates a collection scan. All the documents which are stored in a collection have to be scanned so that only the appropriate documents can be matched. Obviously, this is a highly wasteful tactic as checking each document results in inefficient utilization of resources.

To address this issue, there is a certain feature in MongoDB known as indexes. Indexes perform as a filter so the scanning pool can be shortened and queries can be executed more “efficiently”. Indexes can be categorized as a “special” type of data structures.

Indexes save parts of a collection’s information (data). What they do is that they save a single or multiple fields’ value. The processing of an index’s content is done order-wise.

By default, MongoDB generates an index for the _id field whenever a collection is built. This index is unique. Due to the presence of this index, it is not possible to insert multiple documents which carry the exact same _id field value. Moreover, unlike other indexes, this index is un-droppable.

How to Create an Index?

Open your Mongo Shell and employ the method “db.collection.createIndex()” for the generation of an index. For the complete format, consider the following.

db.collection.createIndex( <key and index type specification>, <options> )

To develop our own index, let’s suppose we have a field for employee name as “ename”. We can generate an index on it.

db.employee.createIndex( { ename: -1 } )

Types

Indexes are classified in the following categories.

  • Single Field
  • Compound Index
  • Multikey
  • Text Indexes
  • 2dsphere Indexes
  • geoHaystack Indexes

 

Single Field Indexes

A single-field index is the simplest index of all. As the name suggests, it applies indexing on a single field. We begin our single-field example with a collection “student”. Now, this student collection carries documents like this:

{

“_id”: ObjectId(“681b13b5bc3446894d86cd342”),

“name”: Adam,

“marks”: 400,

“address”: { state: “TX”, city: “Fort Worth” }

}

To generate an index on the “marks” field, we can write the following query.

db.student.createIndex( { marks: 1 } )

We have now successfully generated an index which operates via an ascending order. This order is marked by the value of an index. With “1” as a value, you can define an index which arranges its contents by using the ascending order. On the other hand, a “-1” value defined an index by using the descending order. This index can now work with other queries that involve the use of “marks”. Some of their examples are:

db.student.find( { marks: 2 } )

db.student.find( { marks: { $gt: 5 } } )

In the second query, you might have noticed “$gt”. $gt is a MongoDB operator which translates to “greater than”. Similar operators are $gte (greater than and equal to), $lt (less than), and $lte (less than and equal to). In our upcoming examples, we are going to use these operators heavily. These are used for filtering out documents by specifying limits.

It is possible to apply indexing on the embedded documents too. This indexing requires the use of dot notation for the embedded documents. Continuing our “student” example,

{

“_id”: ObjectId(“681b13b5bc3446894d86cd342”),

“marks”: 500,

“address”: { state: “Virginia”, city: “Fairfax” }

}

We can apply indexing on the address.state field.

db.student.createIndex( { “address.state”: 1 } )

Whenever queries involving “address.state” are employed by the users, this index would support them. For instance,

db.student.find( { “address.state”: “FL” } )

db.student.find( { “address.city”: “Chicago”, “address.state”: “IL” } )

Likewise, it is possible to build indexes on the complete embedded document.

Suppose you have a collection “users” which contains the following data.

{

“_id”: ObjectId(“681d15b5be344699d86cd567”),

“gender”: “male”,

“education”: { high school: “ABC School”, college: “XYZ University” }

}

If you are familiar with MongoDB, then you know that “education” field is what we call an “embedded document”. This document contains two fields: high school and college. Now to apply indexing on the complete document, we can write the following.

db.users.createIndex( { education: 1 } )

This index can be used by queries like the following.

db.users.find( { education: { college: “XYZ University”, high_school: “ABC School” } } )

Compound Indexes

So far, we have only used a single field for indexing. However, MongoDB also supports the usage of multiple fields in an index. Such indexes are referred to as compound indexes. Bear in mind that there can be no more than 32 fields in compound indexes. To generate such an index, you have to follow this format where ‘f’ refers to the field name and ‘t’ refers to the index type.

db.collection.createIndex( { <f1>: <t1>, <f2>: <t2>, … } )

Suppose we have a collection “items” which stores these details.

{

“_id”: ObjectId(…),

“name”: “mouse”,

“category”: [“computer”, “hardware”],

“address”: “3rd Street Store”,

“quantity”: 80,

}

Compound index can be now applied on the “name” and “quantity” fields.

db.items.createIndex( { “name”: 1, “quantity”: 1 } )

Bear in mind that the order of fields in a compound index is crucial. The index will process by first referencing to the documents which are sorted according to the “name” field. Afterward, it will process the “quantity” field with the values of the sorted “name”.

Compound indexes are not only useful in supporting queries, which equal the index fields, but they also work with matched queries for the index field’s prefix. This means that the index works with queries that have only the “name” field as well as those that have the “quantity” field. For instance,

db.items.find( { name: “mouse” } )

db.items.find( { name: “mouse”, quantity: { $gt: 10 } } )

So far we have been using descending and ascending order with queries. Now, there is no issue in running it with single-field indexes but for the compound indexes, you have to analyze if your queries will work or not. For example, we have a collection “records” which stores documents having the fields “date” and “item”. When queries are used with this collection, then firstly, results are generated by arranging “item” in ascending order, and then a descending order is applied on the “date” values. For instance,

db.records.find().sort( { item: 1, date: -1 } )

Queries where we apply a descending order on the “item” and an ascending order on the “date” value work like:

db.records.find().sort( { item: -1, date: 1 } )

These sort operations can perfectly work with the queries like these:

db.records.createIndex( { “item” : 1, “date” : -1 } )

However, the point to note is that you cannot apply ascending order on both fields like the following.

db.records.find().sort( {“item”: 1, date: 1 } )

 

 

 

 

Design Patterns for Functional Programming


In software circles, a design pattern is a methodology and documented approach to a problem and its solution which is bound to be found repeatedly in several projects as a tumbling block. Software engineers customize these patterns according to their problem and form a solution for their respective applications. Patterns follow a formal structure to explain a problem and then go over a proposed answer as well as key points which are related to either the problem or the solution. A good pattern is one which is well known in the industry and used by the IT masses. For functional programming, there are several popular design patterns. Let’s go over some of these.

Monad

Monad is a design pattern which takes several functions and integrates them as a single function. It can be seen as a type of combinatory and is a core component of functional programming. In monad, a value is wrapped in a box which is then unwrapped and a function is passed to use the wrapped value.

To go into more technicalities, a monad can be classified into running on three basic principles.

·         A parameterized type M<T>

According to this rule, T can possess any type like String, Integer, hence it is optional.

·         A unit function T -> M<T>

According to this rule, there can be a function taking a type and its processing may return “Optional”. For instance, Optional.of(String) returns Optional<String>.

·         A bind operation: M<T> bind T -> M <U> = M<U>

According to this rule which is also known as showed operator due to the symbol >>==. For the monad, the bind operator is called. For instance, Optional<Integer>. Now this takes a lambda or function as an argument for instance like (Integer -> Optional<String> and returns and processes a Monad which has a different type.

Persistent Data Structures

In computer science, there is a concept known as a persistent data structure. Persistent data structure at their essence work like normal data structure but they preserve their older versions after modification. This means that these data structures are inherently immutable because apparently, the operations performed in such structures do not modify the structure in place. Persistent data structures are divided into three types:

  • When all the versions of a data structure can be accessed and only the latest version can be changed, then it is a partially persistent data structure.
  • When all the versions of a data structure can be accessed as well as changed, then it is a fully persistent data structure.
  • Sometimes due to a merge operation, a new version can be generated from two prior versions; such type of data structure is known as confluently persistent.

For data structure which does not show any persistence, the term “ephemeral” is used.

As you may have figured out by now, since persistent data structures enforce immutability, they are used heavily in functional programming. You can find persistent data structure implementations in all major functional programming language. For instance, in JavaScript Immutable.js is a library which is used for implementing persistent data structures. For example,

import { MapD } from ‘immutable’;

let employee = Map({

employeeName: ‘Brad’,

age: 27

});

employee.employeeName; // -> undefined

employee.get(’employeeName’); // -> ‘Brad’

Functors

In programming, containers are used to store data without assigning any method or properties to them. We just put a value inside a container which is then passed with the help of functional programming. A container only has to safely store the value and provide it to the developer in need. However, the values inside them cannot be modified. In functional programming, these containers provide a good advantage because they help with forming the foundation of functional construct and assist with asynchronous actions and pure functional error handling.

So why are we talking about containers? Because functors are a unique type of container. Functors are those containers which are coded with “map” function.

Among the simplest type of containers, we have arrays. Let’s see the following line in JavaScript.

const a1 = [10, 20,30, 40, 50];

Now to see a value of it, we can write.

Const x=y[1];

In functional, the array cannot be changed like.

a1.push(90)

However, new arrays can be created from an existing array. An array is theoretically a function. Technically, whenever a unary function is mapped with a container, then it is a functor. Here ‘mapped’ means that the container is used with a special function which is then applied to a unary function. For arrays, the map function is the special function. A map function processes the contents of an array and performs a special function for all the elements of the element step-by-step after which it responds with another array.

Zipper

A zipper is a design pattern which is used for the representation of an aggregate data structure. Such a pattern is good for codes where arbitrarily traversal is common and the contents can be modified, therefore it is usually used in purely functional programming environments. The concept of Zipper dates back to 1997 where Gérard Huet introduced a “gap buffer” strategy.

Zipper is a general concept and can be customized according to data structures like trees and lists. It is especially convenient for data structures which used recursion. When used with zipper, these data structure are known as “a list with zipper” or “a tree with zipper” for making it apparent that their implementation makes use of zipper pattern.

In simple terms, zipper with data structure has a hole. They are used for the manipulation and traversal in data structures where the hole indicates the present focus for the traversal. Zipper facilitates developers to easily move within the data structure.

Best Functional Programming Practices – When to Use Functional?


For any paradigm, the global developer community experiences several common issues in their development of projects. To counter the recurring issues, they begin exercising certain practices for getting the most out of a paradigm. For functional programming in Java too, there have been a number of practices which have been deemed as useful and valuable for programmers. Let’s go over some of them.

Default Methods

Functional interfaces remain “functional” even if default methods are added. Though, if more than one abstract method is added, then they are no longer a functional interface.

 

 

 

 

@FunctionalInterface

public interface Test {

String A();

default void defaultA() {}

}

As long as the abstract methods of functional interface retain identical signatures, they can be extended by other functional interfaces. For instance,

 

 

 

 

 

 

 

 

 

 

 

 

@FunctionalInterface

public interface TestExtended extends B, C {}

@FunctionalInterface

public interface B {

String A();

default void defaultA() {}

}

 

@FunctionalInterface

public interface C {

String A();

default void defaultB () {}

}

 

 

When usually interfaces are extended, they encounter certain issues. They are recurrent with functional interfaces too when they run with default methods. For instance, if the interfaces B and C have a default method known as defaultD()  then you may get the following error.

interface Test inherits unrelated defaults for defaultD() from types C and D…

To solve this error, the defaultD() method can be overridden with the Test interface.

It should be noted that from the software architecture perspective, the use of a lot of default methods in an interface is detrimental and discouraged. This is something which should be used for updating older interfaces while escaping any backward compatibility issue.

Method References

Many times, methods which were implemented before are called out by lambdas. Hence, for such cases, it is good to make use of method references, a new feature in Java 8.

For example, if we have the following lambda expression.

x -> x.toUpperCase();

Then it can be replaced by:

String::toUpperCase;

Now, this type of code does not only reduce line of codes but it is also quite readable.

Effectively Final

Whenever a variable is accessed which is not “final” and resides in lambda expression, then an error is likely to be caused. This is where “effectively final” comes into play. When a variable is only assigned once then the compiler thinks of it as a final variable. There is nothing wrong in using these types of variables in lambda expressions as their state is managed by the compiler and it can notify for an error as soon as their state is meddled with. For instance, the following code cannot work.

public void A(){

String lVar = “localvariable”;

Test test = parameter -> {

String lVar = parameter;

return lVar;

};

}

In return, the compiler may notify you that “lVar” does not need to be defined because it already has been in the scope.

No Mutation for Object Variables

Lambda expressions are predominantly used in parallelism or parallel computing because of their protection for threads. The paradigm “effectively final” can help at times but sometimes it is not good enough. An object’s value cannot be changed from the closing scope by lambdas. On the other hand, with mutable object variables, it is possible for a state in lambda expression to be modified. For instance check the following.

int[] n = new int[1];

Runnable rn = () -> n[0]++;

rn.run();

Now the above code is perfectly legal because the “n” variable stays “effectively final”. However, it has referenced an object and the state of that object can change. Hence, use this example to remember not writing code which may give rise to mutations.

When to Use Functional?

Before learning functional programming, you must be curious about its actual advantage over other paradigms. When you have a task at hand where you are dabbling with parallelism and concurrency, in such cases functional programming can be a good choice. In real life scenarios, for this purpose Erlang was used a lot in Erricson for its telecommunication work. Likewise, Whatsapp has always been involved in a similar use. Other success stories include the reputable Lucent.

For any individual who has dialed a number in the past three decades in US, there are strong possibilities of their use of devices which have code in a language known as Pdiff. Pdiff itself was created from a functional programming language, Standard ML.

Pdiff’s example can be used to recall functional programming’s brilliance with DSLs (domain specific languages). Sometimes, common programming languages like C++, C#, and Java struggle to create a solution for certain issues where DSLs were the life-savers. While DSLs are not used to design entire systems but they can prove invaluable to code one or two modules. Industry experts consider functional programming as an excellent option to write DSL.

Moreover, functional programming is quite good at solving algorithms, particularly those filled with mathematics. Mathematical problems can be solved well in functional, perhaps due to its closeness in theoretical foundations with mathematics.

When to Not Use Functional?

So when to not use functional programming? It is said that functional programming does not work well with the general “library glue code”. It is a disaster for recipe with the general building of structure classes which are used in mainstream development. This means that in case your code-base is filled with classes working like structures, and if the properties of your object are changing continuously, then functional is probably not the best idea.

Likewise, functional is also not good for GUIs (graphical user interface). The reason is that GUIs have always been deemed more suitable for OOP because of the reusability factor. In GUI applications, modules are derived with little changes from other modules. There is also the “state” factor as GUIs are stateful (at least in the view).

 

Dos and Don’ts of Functional Programming


While working with functional programming in Java, there are several do’s and don’ts. Let’s go over the most common ones. By making use of these tips, you can improve your functional programming code-base with good results.

Standard Functional Interfaces

The Functional interfaces in “java.util.function” are good enough to fulfill requirements for method references and lambda’s target types. As explained earlier, these interfaces are abstract, thereby making it easier to customize them freely into a lambda expression. Hence, before writing new functional interfaces, makes sure to check this package which may satisfy your requirements.

Suppose you have in interface FuncInt

@FunctionalInterface

public interface FuncInt {

String f1(String txt);

}

Now you have a class for using the interface with the name useFuncInt which has a method for sum.

public String sum(String txt, FuncInt) {

return FuncInt f1(txt);

}

For execution of it, you would have to add the following:

FuncInt fi = txt -> txt + ” coming from lambda”;

String output = useFuncInt.add(“Message “, useFuncInt);

Now, if you observe closely, you will realize that FuncInt is a function which takes one argument and returns with an answer. With the release of Java 8, this functionality is enabled through the interface in “Function<T,R> which is entailed in the package of “java.util.function”. Our FuncInt can be fully eliminated and our code can be modified to.

public String sum(String txt, Function<String, String> f) {

return f.apply(txt);

}

For execution, write the following:

Function<String, String> f =

parameter -> parameter + ” coming from lambda”;

String output = useFuncInt.sum(“Message “, f);

Annotation Matters

Make the use of annotation by adding functional interfaces with the help of @FunctionalInterface. If you are unfamiliar with annotations, then put it with a tag which is used to represent the metadata; this could be related to an interface, method, field, or class about any useful information that can prove to be helpful for the JVM (Java Virtual Machine) or the complier.

The @FunctionalInterface may not seem logical at the first glance because without adding it, the interface would work fine as a functional one as long it contains only one abstract method.

However, in large-scale projects where the increasing number of interfaces may overwhelm you, manual control is tricky and exhaustive; it becomes a nightmare to keep track of it. Sometimes, an interface which was created as a functional interface may get additional abstract methods, therefore ceasing to exist as a functional interface. This is where the @FunctionalInterface annotation works like a charm. By using it the complier would display an error whenever there is any intrusion by a programmer to change the structure of a functional interface and disrupt its “purity”. Likewise, while working in teams, it can be helpful for other coders to understand your code and identify functional interfaces easily.

Hence, always use something like this,

@FunctionalInterface

public interface FuncInt{

String printSomething();

}

Instead of writing something like this:

public interface FuncInt{

String printSomething();

}

Even if you are working on beginner projects, make it a habit to always add the annotation or you may face the repercussions in future.

Lambda Expressions Cannot Be Used Like Inner Classes

When an inner class is used, “scope” is generated. This means that we can add variables which are local with the enclosing scope after instantiating local variables having identical names. The keyword “this” can also be used in our class for referencing to its own instance.

With lambda expressions, we work by using enclosing scope. This means that variables cannot be overwritten in the body of the lambda expression from the enclosing scope. We can use “this” for referencing.

For instance, for our class UseTest, we have “test” as our instance variable:

private String txt = “Encircling the scope”

Now, you can use another method of our class with the following code and run it like this:

public String example()

{

Test test = new Test();

String mssg = “inner value of the class”

@Override

Public String returnSomething(String txt)

Return this.mssg;

}

};

String output = test.returnSomething(“”);

Test testLambda = parameter -> {

String mssg = “Lambda Message”;

return this.mssg;

};

String outputLambda= testLambda.returnSomething(“”);

return “Output: output = “ + output + “outputLambda = “ + outputLambda;

When you will run the “example” method, you may get the following: “Output: output =  inner value of the class, outputLambda = Encircling the scope

This means that you can access and use a local variable by accessing its instance. By using “this”, you might have used the variable “test” of the UseTest class but you were unable to get the “test” value which in entailed in the body of the lambda.

Lines of Code in Lambdas

Ideally, lambda expressions should comprise of a single line of code because this makes it a self-explanatory concrete implementation where the clear execution of an action at some data is represented.

If you require more lines for your functionality, then write something like this:

Test test = parameter -> generateText(txt);

private String generateText(String txt) {

String output = “Some Text” + txt;

// several lines of code

return output;

}

Refrain from writing something like this:

Test test = txt -> { String output= “Some Text” + txt;

// several lines of code

return output;

};

Though, it is important to note that sometimes, it is ok to have more than one line of lambda expressions where the use of another method may prove to be counter-productive.

Type of Parameters

The compilers are powerful enough to ascertain the types of parameters in lambda expressions with type inference. Hence, there is no need for adding the parameter type explicitly.

Don’t write something like this.

(String x, String y) -> x.toUpperCase() + y.toUpperCase();

Instead, write something like this,

(x, y) -> x.toUpperCase() + y.toUpperCase();

Java Lambdas


So far we have talked a lot about functional programming. We discussed the basics and even experimented with some coding of functional interfaces. Now is the right time to touch one of the most popular features of Java for functional paradigm, known as lambda expressions or simply lambdas.

What Is a Lambda Expression?

A lambda expression provides functionality for one or more functional interface’s instances with “concrete implementations”. Lambdas do not require the use of a class for their use. Importantly, these expressions can be viewed and worked by coding them as objects. This means that, like objects, it is possible to pass or run a lambda expression. The basic style for writing a lambda expression requires the use of an “arrow”. See below:

parameter à the expression body

On the left side, we have a “parameter”. We can write single or multiple parameters for our program. Likewise, it is not mandatory to specify the parameter type because compilers already ascertain the parameter type. If you are using a single parameter, then you may or may not add a round bracket.

However, if you intend to add multiple parameters, then make sure to use round brackets (). Sometimes, there is no need of parameters in a lambda expression. For such cases, it is possible to signify them by simple adding an unfilled round bracket. To avoid error, use round brackets for parameter whether you are using a 0, 1, or more parameters.

On the right side of the lambda expression, we can have an expression. This expression is entailed in curly brackets. Like parameters, you do not require brackets for a single expression while multiple expressions require one. However, unlike parameter, the return type of a function can be signified by the body expression.

Without Lambdas

To understand lambdas, check this simple example.

package fp;

public class withoutLambdas {

public static void main(String[] args) {

withoutLambdas wl = new withoutLambdas(); // generating instance for our object

String lText2 = “Working without lambda expressions”; // here we assign a string for the object’s method as a parameter

wl.printing(lText2);

}

public void printing(String lText) { // initializing a string

System.out.println(lText);              // creating a method to print the String

}

}

 

The output of the program is “Working without lambda expressions”. Now if you are familiar with OOP, then you can understand how the caller was unaware of the method’s implementation i.e. it was hidden from it. What is happening here is that the caller gets a variable which is then used by the “printing” method. This means we are dealing with a side effect here—a concept we explained in our previous posts.

Now let’s see another program in which we go one step ahead, from a variable to a behavior.

package fp;

public class withoutLambdas2 {

interface printingInfo {

void letsPrint(String someText);  //a functional interface

}

public void printingInfo2(String lText, printingInfo pi) {

pi.letsPrint(lText);

 

}

 

public static void main(String[] args) {

withoutLambdas2 wl2 = new withoutLambdas2(); // initializing instance

String lText = “So this is what a lambda expression is”; // Setting a value for the variable

printingInfo pi = new printingInfo() {

@Override // annotation for overriding and introducing new behavior for our interface method

public void letsPrint(String someText) {

System.out.println(someText);

}

};

wl2.printingInfo2 (lText, pi);

}

 

 

}

In this example the actual work to print the text was completed by the interface. We basically formulated and designed the code for our interface’s implementation. Now let’s use Lambdas to see how they provide an advantage.

 

package fp;

public class firstLambda {

 

interface printingInfo {

void letsPrint(String someText);    //a functional interface

}

public void printingInfo2(String lText, printingInfo pi) {

pi.letsPrint(lText);

}

 

public static void main(String[] args) {

firstLambda fl = new firstLambda();

String lText = “This is what Lambda expressions are”;

printingInfo pi = (String letsPrint)->{System.out.println(letsPrint);};

fl.printingInfo2(lText, pi);

}

See how we improved the code by integrating a line of lambda expression. As a result, we are able to remove the side effect too. What the expression did was use the parameter and processed it to generate a response. The expression after the arrow is what we call as a “concrete implementation”.

Core Concepts of Functional Programming


Now that you have learned about the paradigm shift to functional programming, let’s go into the depths of the fundamentals concepts that power functional programming. The comprehension of these basic concepts is important to create high-quality functional programming applications. You may be tempted to directly begin coding but these concepts can help you become a better coder.

1- Pure Functions

In functional programming, everything is seen as a function. Each function has to be “pure”. Pure here refers to two basic capabilities:

No Side Effects

A function can never be pure if it carries even a single side effect. A side effect is a property when a function’s states are modified by other functions. By states, we mean the data like variables or data structures. Pure functions do not carry any side effects; hence their memory or I/O operations can’t be affected. Now, you might wonder that why exactly does their presence considered bad. Well, because they make functions “unpredictable” where a function has to rely on its system’s state.

On the contrary, if a function’s state cannot be changed, then the same output is generated for the given input. A side effect of a function can also mean to write any operation which has been applied to the disk or turning on/off a control of your front-end UI’s function.

Same Result with Multiple Calls

Whenever a function is called without any modifications in its arguments, the same results are generated. Consider an example where you have designed a simple function “multiply (4, 5). Now, this function is expected to generate the same results .i.e. for each invocation. However, if you were programming in other paradigms then random functions or global variables may not allow your result to remain the same.

Pure functions also offer “memoisation”. Memeoisation refers to a technique in which pure functions’ output (always same result) is saved in the cache memory. Now, whenever such functions are invoked, caching helps to enhance the performance and speed of the application.

2- Higher Order Function

The concept of a function which is higher order is known in mathematics as well as computer science. Generally, they possess two fundamental characteristics.

Return Type

The return type of a higher-order function has to be a function. For example, review the following code in Java.

package fp;

 

public class higherOrderfunction {

public static void main(String args[]) {

 

System.out.println(A(4));

 

}

static int marks() {

int a=5;

return a;

}

static int A(int total) {

total =4;

return total+marks();

}

}

To simplify things, we have constructed a simple function marks. This function holds an integer value “a” which is returned. Now, we have a function A. A takes an argument for an integer total and assigns it a value. Now, comes the actual part. See how in return, we have used marks method as a return type. Since we have processed our function by returning another function; hence this function is a higher-order function.

Arguments

Another characteristic of a higher-order function can be its use of functions as input parameters. For instance, see this see pseudo-code:

Public areaRect (lb) // Here areaRect represents a function that takes arguments from another function lb to calculate length  and breadth

{

int area;

area =l*b;

return lb; //

}

This function is a higher-order function because it processes itself using another function as a parameter.

3- First Class Functions

After higher-order functions, we have first-class functions. These are not too dissimilar to higher-order functions. It is important to note that a first-class function always adheres to the terms and conditions of a higher-order function. So what this means is that a first-class function has to return another function as well as contain a parameter in the form of function. Hence, by default, all first-class functions are higher-order functions. So what exactly is the difference between them? Well, context matters!

By referring a language to have support for first-class functions, we generally mean that uses its functions as values that can be easily passed around.

On the other hand, the term higher-order is more associated with the mathematics outlook when pure problem-solving requires a more theoretical and general perspective of the problem.

4- Evaluation

Some languages support strict evaluation while others offer non-strict evaluation support. This evaluation is targeted at the language’s processing of an expression while considering the function parameters or arguments. To familiarize yourself better with the concept, check this simple example:

Print length([4-3,4+6,1/0,7*7,3-8])

When a programming language uses non-strict evaluation to process this expression, then it simply returns back a value of 5 .i.e. the total number of elements. Such evaluation does not concern itself with the depth of values.

On the other hand, the same expression returns an error with strict evaluation because it found the third element “1/0” to be incorrect. Hence, this means that strict evaluation is more stringent and processes an expression more deeply.

In a few scenarios, non-strict evaluation has to enforce processing of strict evaluation when a function needs to be evaluated on a “stricter” basis due to an invocation.

5- Referential Transparency

In functional programming, the usual assignment of values is not offered. Variables are immutable; a value defined once is the final one which is not possible to be changed in future. It is the property which makes them without any side-effects. To understand further, consider the following example, where the value of x is changed after each evaluation.

x=x*15

In the start of the program, x was assigned a value of 1. The first evaluation made it 15.

Now, in the second evaluation, the value of x changes to 225.

Now, this function is not referentially transparent because the value of x is continuously changing.

So now, if we go by the functional concepts, then our example can be altered into this pseudocode:

int sum(int x)

{

x=x+2;

}

This type of function ensures that the value of x remains constant and it cannot be altered implicitly.