Month: January 2019

by

What Is Virtualization? What Are the Types of Virtualization


Recently I was working on a POC which triggered a very intriguing topic -Virtualization and how it is changing IT culture.

Virtualization is written on the blue sky. As the name suggests, virtualization involves producing a “virtual” environment. This environment is implemented on a server to execute the required programs without any complexity in interrupting other platforms and applications of the server. For instance, in a traditional architecture you can run a single application which runs on an operating system configured on a server. Through virtualization, a new layer is added to the top of the server which can host multiple OS having their own applications.

To understand virtualization, you must also know what a VM or virtual machine is. A VM is a heavily isolated container which encompasses an application and OS. All the VMs are independent. VMs are stored on a single device to run multiple applications and OS at once through a single host.

The software which is used to link the host machine and the virtual machine is commonly referred to as a hypervisor. It is the hypervisor which serves as an intermediary between the virtual environment and the hardware equipment of a server and is responsible to properly allocate resources like memory and number of cores.

Types of Virtualization

Virtualization can be applied to multiple components in the IT infrastructure of an organization. Following are the fundamental types of virtualization.

  • Server Virtualization

Server virtualization is also known as hardware virtualization. The idea behind such virtualization is to break down a server “logically” into multiple segments of hardware which effectively work as virtual servers. All of these smaller servers are powered to host their own virtual machines.

However, the complete set of servers would be considered as a single one whenever a process comes up with a request related to hardware. Such a virtualization technique is useful in boosting the processing capabilities of the organization without spending on a new server. Likewise, it can also improve the uptime and enhance the resource utilization of the hardware.

  • Storage Virtualization

In storage virtualization, all of the storage devices are lumped or consolidated together, so they are “visible” and function as a stand-alone device. This is done in order to achieve homogenization to get increased speed and capacity, load balancing, enhanced optimization, and decreased downtime—factors which often plague organizations when they work with several different storage devices.

You can observe the use of storage virtualization whenever you are attempting to create more than a single partition from out of your hard drive. When several storage devices are combined in a single device, then the virtualization is more commonly known as block virtualization.

  • Memory Virtualization

This type of virtualization collects and combines the physical memory from all servers and accumulates it into a virtualized pool. This is beneficial because it allows expanding the contiguous working memory.

Operating systems often make use of such techniques. For instance, Microsoft’s operating system facilitates some part of the storage to act as the RAM’s extension. This is an OS level memory virtualization where the OS offers the memory pool. For the application level, applications offer the memory pool.

  • Software Virtualization

In software virtualization, several virtual environments are generated which are then run on the host machine. Such virtualization allows the guest OS to function. For instance, if a host machine is natively running on Windows 10, then software virtualization can help to run Android 8.0 on the machine where it makes use of the exact hardware resources which are granted to the host machine.

Software virtualization is subdivided into OS virtualization, application virtualization, and service virtualization. In application virtualization, you can host many applications in the virtualized space. In service virtualization, you can host an application services and processes in the virtualized space.

  • Network Virtualization

Such a virtualization solution requires dividing several mini-networks from a single physical network. These mini-networks are configured to communicate with each other; it is possible to completely cut off any communication between them.

This virtualization is effective because it supports the limitation of file transfer in various networks. Moreover, it is a secureoption because instead of the whole network, only a single mini-network is breached when the company is hit by a cyberattack like man-in-the-middle attack.

Similarly, it enables a greater level of monitoring and facilitates networking professionals to identify the usage of data. As a result, a greater degree of reliability is attained as a problem in one network cannot necessarily disrupt another network.

  • Desktop Virtualization

This virtualization is routinely used by organizations to improve productivity. In desktop virtualization, a remote server stores users’ desktop environment, thereby authorizing them to work from anywhere in the world. This way, employees can work on an urgent-basis, whether they are on a vacation or when they are on a sick leave. Hence, organizations gain a huge advantage and are able to maximize their revenues.

  • Data Virtualization

When organizations use various data sources like MongoDB, SQL Server, and other DB products, then they find integration issues. To address, such issues they make use of data virtualization for aggregating data efficiently.

Data virtualization refers to an approach in which data is gathered from multiple data sources in order to design a single, virtual, and logical perspective or view. Such a view maybe used by a dashboard for an analytics tool or a software application. Data virtualization provides a layer of abstraction on storage data for location, storage structure, access language, and application programming interface.

  • Service Virtualization

When organizations supports parallel development lots of time service provider becomes bottleneck for consumers. Even existing service can become bottleneck for consumer due to availability,connectivity. Service Virtualization refers to the approach to emulate the behavior of the original component by creating similar response and request structure. This helps organization in faster and independent development and testing.

Service Virtualization is different from Mock service.Mock service is more of static request response implement vs full implementation in service virtualization. Also,Mock service are stateless and context anware whereas service virtualization can be stateful and state aware.

Final Thoughts

With the emergence and advancements of cloud computing, virtualization has changed the game for many organizations. Companies use it as one of the major sections of their disaster recovery and backup strategy. Whenever, a disaster like a cyberattack, power outages, or any similar issues arises, virtualization simplifies and eases up and ensures to quicken the recovery period. Similarly, it is beneficial to effectively allocate budget for hardware, and therefore saves from unnecessaryinvestments in servers. This effect is particularly positively felt in SMEs (small and medium enterprises).

Today, by limiting the wastage of computing equipment and minimizing costs, virtualization has truly become a key IT component of any successful organization.

Advertisement

How Has Google Improved Its Data Center Management Through Artificial Intelligence


Historically, the staff at data centers adjusted the settings of the cooling systems to save energy costs. Times have changed, and this is the sweet age of AI where intelligent systems are on guard 24/7 and automatically adjust these settings to save costs.

Last year, a tornado watch prompted Google’s AI system to take control of its cooling plant in a data center and it modified the system settings. The staff at Google was initially perplexed because the changes did not make sense at the time. However, after a closer inspection, the AI system was found to be taking a course of action that reduced the energy consumption.

The increase and decrease in temperature, humidity levels, and atmospheric pressure force the change in weather conditions, and they can stir a storm. This weather data is used by Google’s AI to adjust the cooling system accordingly.

Joe Kava, Google’s Vice President of data centers, revealed Google’s use of AI for data centers back in 2014. At that time, Kava explained that the company designed a neural network to assess the data which is collected from its data centers and suggested a few strategies to enhance its processing. These suggestions were later utilized as a recommendation engine.

Kava explained that they had a single solution which would provide them with recommendations and suggestions. Afterward, the qualified staff at Google would begin modifying the pumps, heat exchangers, and chillers settings according to the results of AI-based recommendations. In the last four years, Google’s AI usage has evolved beyond Kava’s proposed vision.

Presently, Google is adopting a more aggressive approach. Instead of only dishing out recommendations to the human operators could act on them, the new system would itself go onto adjust the cooling settings. Jim Gao, a data engineer at Google, said that the previous system saved 20 percent energy costs and that the newer update would save up to 40 percent in energy consumption.

Little Adjustments

The tornado watch is only a single real-world instance of Google’s powerful AI and its impact on energy savings to an extent which was impossible with manual processes. While at first glance, the minor adjustments done by the AI-enabled system might not seem enough. However, the sum of each savings results in a huge total.

Kava explains that the detailing performed by the AI systems makes it matchless. For instance, if the temperature in the surroundings of the data center goes from 60 degrees Fahrenheit to 64 degree Fahrenheit while the wet-bulb temperature is unaffected, then an individual from the data center staff would not go think much about updating the settings of the cooling system. However, the AI-based system is not so negligent. Whether there is a difference of 4 degrees or 40 degrees, it keeps on going.

One interesting observation regarding the system was its noticeably improved performance during the launch of new data centers. Generally, new data centers are not efficient as they are unable to get the most of the available capacity.

From Semi to Full Automation

The transfer of critical tasks of the infrastructure to the AI system has its own implications and considerations.

With the increase of data and runtime, the AI system becomes more and more powerful and therefore, management also starts to have faith in the system, enough to give it some control. Kava explained that after some experimentation and results, slowly and gradually the semi-automated tools and equipment are replaced by fully automated tools and equipment.

Uniformity is the key to Google’s AI exploits; it is not possible to implement AI at such a massive scale without uniformity. All the data centers are designed to be distinct such that a single AI system is not possible to be integrated across all of them at the same time.

The cooling system of all the data centers are constructed for maximum optimization according to their geographical locations. Google has tasked its data engineering team to continuously look for any possible techniques for energy savings.

Additionally, ML-based models are trained according to their sites. The models have to be programmed to follow that site’s architecture. This process takes some time. However, Google is positive that this consumption of time would result in better results in the future.

The Fear of Automation

One major discussion point with this rapid AI automation and similar AI-based ventures is the future of “humans” or the replacement of the humans. Are the data center engineers from Google going to lose their jobs? This question contains one of mankind’s biggest fears regarding AI. As AI progresses, this uncertainty has crept into the minds of workers. However, Kava is not worried. Kava stated that Google still has staff at its disposal at data centers that is responsible for maintenance. While AI may have replaced some of their “chores”, the staff still has to perform corrective repairs and preventative maintenance.

Kava also shed some light on some of AI’s weaknesses. For instance, he explained that whenever the AI system finds itself in the midst of uncharted territory, it struggles to choose the best course of action. Therefore, it is unable to mimic the brilliance of humans in making astute observations. Kava concluded that it is recommended to use AI for cooling and other data center related tasks, though he cautioned that there must be some “human” presence to ensure that nothing goes amiss.

Final Thoughts

Google’s vision, planning, and execution of AI in its data centers are promising for other industries too. Gao’s model is believed to be applicable to manufacturing plants that also have similar setups like cooling and heating systems. Similarly, a chemical plant could also take advantage of AI and likewise, a petroleum refinery may use AI in the same way. The actual realization is that, in the end, such AI-based systems can be adopted by other companies to enhance their systems.

The Growing Role of Artificial Intelligence in Data Centers


According to Infosys, more than 75 percent of IT experts view artificial intelligence as a permanent strategic priority which can assist them in innovating their organization’s structure. Infosys’ survey receives credibility from the fact that the AI systems are expected to receive investments worth $57 billion by the year 2021. That being said, the implementation of AI is a complex task which requires considerable time and decision-making to go smoothly. Today, AI has initiated the transformation of all the global industries.

One of such industries is the data center industry where AI is making its mark slowly and gradually. Data centers power the operations of organizations all around the world. The data volumes are increasing daily, putting more and more strain on the hardware and software setups in the organization. Consequently, managers are forced to introduce new servers and hardware equipment so their IT infrastructure becomes powerful enough to store and process data without any issue. Currently, most of the centers are not able to maximize their output because they use legacy systems. So how is AI transforming data centers?

Energy Consumption

Energy consumption remains one of the most critical and dire issues in data centers. Bear in mind that as of now, about 6 percent of the world’s electricity is used by data centers. With the computing requirements climbing up day-by-day, it is fair to assume that the energy consumption of data centers will also increase.

On one hand, companies have to address the cost factor, and on the other hand, global warming is mounting pressure on organizations to do their part and act more ‘responsibly’ towards the environment. Particularly, the data center industry is one of those industries that are viewed negatively by the supporters of green energy.

Some data centers have attempted to address such issues by accepting renewable energy. However, there are qualms about its ineffectiveness for smaller setups. There are few companies that have resorted to AI as the answer to their common problems.

AI is being used for real-time monitoring to reduce energy consumption. Moreover, AI is used for parallel and distributed computing to achieve a greater level of productivity. Some organizations have identified and resolved networking troubleshooting via AI. Similarly, there are those who adjust their heating and cooling mechanisms via AI. Due to the widespread use of artificial intelligence, there is no need for staff members to continuously manage mundane tasks such as setting the office temperature.

Security

Security is also one of the most pressing issues for data centers. Cybercriminals have particularly set their eyes on the data centers. With the amount of sensitive data being stored in these data centers, it is not surprising that hackers try to target these centers. For instance, if a cybercriminal group succeeds in a ransomware attack on a data center then by just locking the servers, they bring the entire organization down on its knees. Dreading the losses due to downtime and reputational damage, the company has no option but to pay a ransom to save their data center from complete destruction. Unfortunately, ransom payment does not guarantee the return of data. While organizations are trying their best to infuse the most effective measures to restrict such attacks, they have found AI as an underrated ally in their proactive action against cyber attacks.

AI’s addition in the equation offers a greater level of flexibility and sophistication to protect the data and minimize the dependence of systems on manual intervention. Unlike humans, AI can be available 24/7 and may become the wall that ultimately safeguards you from a cyber attack. For instance, Darktrace—a British organization—leveraged AI to specify a normal network behavior where cyber threats are assessed and identified on the basis of a deviated activity.

Data Center Staffing

AI is also offering a chance for organizations to reduce their staff shortages so they can assign their qualified personnel to the relevant areas. It is expected that with AI in the mix, the standard tech support responsibilities in the center would be handed over to AI-based systems.  These responsibilities would include automation of routine and mundane tasks like the following:

  • Resolving any incoming issue.
  • Working on the help desk support.
  • Provision of services and resources.

Additionally, AI would provide an edge by capturing new symptoms, events, and scenarios for the generation of a functional knowledge base to aid the external and internal stakeholders to learn from the past issues and avoid repeating the same mistakes in future.

However, there will be times when human intervention would be necessary. In such cases, a connection can be established with senior staff members who can fulfill the required task through their years of experience.

Predictive Analytics

With enhanced outage monitoring, AI is providing a major advantage to data centers. AI systems are able to detect and predict any incoming data outage. They can continuously track the performance of all the servers and assess the storage operations like the utilization of disk.

All of this has been made possible through contemporary predictive analytics tools which do not only increase reliability but also are fairly easy to use. Probably the biggest advantage of predictive analytics is that it supervises the workload through optimization, lessens the burden from systems, and distributes the workload more evenly among all the hardware tools.

This modern outlook of data centers is widely different from the conventional data center practices. Traditionally, such troubleshooting was based completely on manual assistance, research, and computation—computers were merely a tool to execute and command their strategies. AI, on the other hand, positions itself as an independent player which can be seen more as a professional colleague rather than a tool.

Final Thoughts

As the management of data centers becomes tougher and more complex with the passing time, AI has been a welcome entry in the space as an IT technology. AI has improved the overall output without any notable compromise. It remains to be seen what more advancements arrive in data centers in the near future. For the time being, AI has done a marvelous job at managing data centers.

How Does a Decision Tree Function?


In the last post, we discussed how artificial neural networks are modeled on a human brain. There are other algorithms too which have been inspired by the real world. For instance, we have the decision tree algorithm in machine learning which is founded on the basis of a tree. Such an algorithm is used for decision analysis. The algorithm is also frequently used in data mining to derive meaningful results. So, how exactly does it work?

To understand a decision tree, let’s suppose an elementary example in which we have a dataset of passengers of a ship. It is expected that a violent storm would cause the ship to get wrecked. Now the problem at hand is to predict the survival rate of a passenger based on their characteristics. Their attributes (also known as features) are mainly their age, spch (any spouse or children with them), and age.

dtree

As you can see, a decision tree is visually represented in an upside-down approach where instead of placing the root at the top, we present it at the top. The italicized text which shows our condition represents the internal node which divides the three into edges (branches). The branch which is not divided any further is referred to as the decision (leaf).

If you analyze the above example, then you can recognize the fact that all the relevant relations are easily viewable, thereby making for strong feature importance. This approach is also called a “learning decision tree from data”. The tree in our drawn example is categorized under a classification tree as it is used to classify the survival rate or fatality rate of a passenger tree. The other category is known as regression tree which is not too dissimilar, except for the fact that they deal with continuous values. The decision tree algorithms are broadly depicted as CAT— Classification and Regression Trees.

The growth of a decision tree depends upon its features (attributes or characteristics) and the conditions which are used to divide the tree with a clear intent about the stopping point of the three. Often, the growth of tree exceeds to arbitrary levels where some trimming is required for better results.

Recursive Binary Splitting

Recursive binary splitting uses a cost function to test all the features and the split points. For instance, in the above example from the root, all features were analyzed after which groups were formed from the divisions of the training data. Our example has 3 examples which mean we require 3 splits. Subsequently, we are going to compute the cost of each split in terms of accuracy. When the least costly split is discovered, which refers to the sex feature in our example then the feature is chosen. This approach of the algorithm is naturally recursive because more groups can undergo subdivisions by repeating the same process. Therefore, the algorithm falls into the category of greedy algorithms. This also means that the most effective classifier is the root node.

Split Cost

Let’s try to understand cost functions more closely while working with classification and regression. Cost functions always attempt to identify the branches which exhibit similarity. Therefore, it is certain that any input which is test data is bound to adhere to the specific path.

Regression: sum(y-prediction)^2

For instance, consider the real estate industry a problem requires the prediction of house prices. In this case, the decision tree initiates the splitting processing and analyzes all the features from the training data. It calculates the input of training data to generate mean for responses which are treated as a prediction for their respective groups. The function is performed for all the data points while a cost is generated for the candidate splits. In the end, the split which consumed the smallest cost is chosen.

Classification: G = sum(pk * (1 — pk))

To determine the quality of a split, the gini score is used which assesses the mixing of the response classes in the split’s groups. In the above equation, pk refers to the proportion in which a particular group has similar class inputs. Maximum purity of a class is achieved when it established that a group encompasses the same class’ inputs. In such a scenario the value of pk maybe either 0 or 1 while G remains 0. The worst purity is established when a node gets 50-50 split for a group’s classes. In binary classification, the values of pk and G would be 0.5 each in such a scenario.

Putting a Stop to Split

There is a point at which the split of the tree must be stopped. Generally, problems have several features which means that the resulting split is also huge, thereby creating a large tree. This is an undesired scenario because such trees raise over-fitting issues. One strategy for stopping a split is to define the lowest number for training inputs which are to be assigned for all the leaves. For instance, in the above example, we can take 15 passengers to reach a consensus or decision for survival or death whereas any leaf which is bombarded with less than 15 passengers is duly rejected. Conversely, you can also define the max depth for the model. Max depth is the longest path’s total length which exists between a root and a tree.

Pruning is used to enhance the performance of a decision tree. In pruning, any branch with low or weak feature importance is eliminated, thereby minimizing the tree’s complexity and boosting its predictive strength. Pruning can either initiate from the leaves or the root. In simpler scenarios, pruning begins from the leaves where it eliminates nodes that have the most popular class of that leaf unless they are not violating accuracy. This strategy is also called as reduced error pruning.

Final Thoughts

The above-mentioned knowledge is enough to complete your initial understanding of a decision tree. You can begin its coding by using Python’s Scikit-Learn library.

What Is Artificial Neural Network and How Does It Work?


The whole idea behind artificial intelligence is to make a machine act like a human being. While many sub-divisions of AI originated with their own set of algorithms to mimic humans, artificial neural networks (ANNs) are AI at its purest sense; they mimic the working of the human brain, the core and complex foundation which influences and affects the thinking and reasoning of human beings.

What Is an Artificial Neural Network?

ANN is a machine learning algorithm. It is founded on the scientific knowledge about organic neural networks (working of the human brain). ANN works quite similar to how human beings analyze and review information. It is composed of several processing units which are linked together and perform parallel processing for the computation of data.

As machine learning is primarily focused on “learning,” ANNs continuously learn and adapt. The processing units in ANNs are commonly referred to as neurons or nodes. Bear in mind that neuron in biology refers to the most basic units in the human nervous system. Each node is linked via arcs which have their own weight. The artificial neural network is made up of three layers.

Input

The input layer is responsible for accepting explanatory attribute values which are collected from observations. Generally, input nodes are explanatory variables. Patterns are submitted to the network by the input layer. Subsequently, those patterns are then analyzed by the hidden layers. The input layer nodes are not involved in modifying any data. They accept individual values as inputs and then perform duplication of the value so it can be passed on to multiple outputs.

Hidden

The hidden layers modify and transform values collected from the input layer. By utilizing a technique of weight links or connections, the hidden layer initiates computation on the data. The number of hidden layers depends upon the artificial neural network; there may be one or more than one hidden layers. Nodes in this layer multiply the collected values by the weights. Weights are a predetermined set of numbers which convert the input values with the help of summation to generate an output in the form of a number.

Output

Afterward, the hidden layers are connected to an output layer which may also receive a connection directly from an input layer. It generates a result, which is associated with the response variable’s prediction. Generally, when the machine learning process is geared towards classification and its disciplines, there is a single output node. The collected data in the layer is integrated and modified for the generation of new values.

The structure of a neural network is also called topology or architecture. All the above layers of the ANN form the structure. The planned design of the structure bears utmost importance to the final findings of the ANN. At its most basic, a structure is divided into two layers which are comprised of one unit each.

The output unit also possesses two functions: combination and transfer. When there are multiple output units, then logistic or linear regression can be at work and the nature of the function ultimately decides it. ANN’s weights are actually coefficients (regression).

So what do the hidden layers do? Well, the hidden layers are incorporated into ANNs to enhance the prediction strength. However, it is recommended to add them smartly because excessive use of these layers may mean that the neural network stores all the learning data and may not able to generalize, causing an over-fitting problem. Over-fitting arises when the neural network is not able to discover patterns and is heavily reliant on its learning set to function.

ai1

 

Applications

Due to their accurate predictions, ANNs have broad adoption across multiple industries.

Marketing

Modern marketing focuses on segmenting customers within well-defined and distinct groups. Each of these groups exhibits certain characters that are reflecting of its customer habits. In order to generate such segmentation, neural networks present themselves as an efficient solution for predicting strength to identify patterns in a customer’s purchasing habits.

For instance, it can analyze how much time customers take between each purchase, how much do they spend, and what do they mostly purchase. ANN’s input layer takes all the attributes like location, demographics, and other personal or financial information about a customer to generate meaningful output.

Supervised neural networks are usually trained to comprehend the link between clusters of data. On the other hand, unsupervised neural networks are used for segmentation of customers.

Forecasting

Forecasting is a part and parcel of a varied list of domains including governments, sales, finance, and other industries, especially their use in the monetary and economic aspects. Often, forecasting faces a tumbling roadblock because of its complexity. For instance, the prediction of stocks is considered difficult because the stock market addresses multiple seen and unseen factors where traditional forecasting becomes ineffective.

This conventional forecasting is founded merely on statistics. ANNs use the same statistical methods and techniques and enhances forecasting where its layers are sophisticated enough to tackle the complexity of the stock market. Moreover, in contrast to the conventional methods, ANN is non-restrictive for input values and residual distributions.

Image Processing

Since the layers in artificial neural networks are able to accept several input values and compute them flexibly to determine complex and non-linear hidden relationships, they are well-equipped to serve in image processing and character recognition. In criminal proceedings like bank frauds, fraud detection requires accurate results for character recognition because humans cannot go over thousands of samples to pinpoint a match. Here, ANNs are useful as they are able to recognize the smallest of irregularities. Similarly, ANN is used in facial recognition with positive results where they are able to improve governance and security.

Final Thoughts

The emergence of artificial neural networks has opened a whole new world of possibilities for machine learning. With their adoption in real-world industries, the algorithm has become one of the most trending and research topics in a short period of time.

by

How Is Machine Learning Assisting Organizations to Tackle Sophisticated Cyberattacks?


Previously, cybercriminals were limited in their approach. With the passage of time, they evolved and firmed their grasp on newer technologies. As a result, they were able to initiate highly sophisticated campaigns against businesses and individuals alike. One such example is of the attack on LapCorps—one of the prominent names in the healthcare industry in the USA.

Over the past few years, the trend has worsened as cybercriminals are directly challenging governments through attacks in cities and town governments. For instance, just a few months ago, the American cities of Atlanta and Baltimore faced city-wide cyberattacks that halted their public services. What’s more worrisome is the fact that authorities have discovered that some of the cyber attacks were backed by other countries, thereby changing the face of modern-day warfare to cyber warfare.

In such challenging times in the cybersecurity industry, the recent advancements in machine learning have made it highly useful against cyber attacks. The entire purpose of machine learning is to “learn” from the past and update itself with the passage of time. This vision is perfectly suited to address cyber attacks where machine learning can learn from the historical data of cyber attacks like the information of their victims, their target industries, their patterns, and other related information and can then use it to prevent any future attacks while evolving at the same time. Following are some of the cases where machine learning has been pretty impressive against some major threats.

Classification

Traditionally, burglars and robbers used to analyze and research targets and carry out crimes accordingly. Today, the situation is the same but the battleground is different as criminals have transformed into cybercriminals. These cybercriminals target specific businesses or persons to infect their servers with a technique called spear phishing.

In order to combat these cyber attacks, several phishing detection solutions have been released albeit with limited success because they do not fare well on the precision and quickness of their actions against such infections. As a consequence, users are left alone to fight off cyber attacks.

Machine learning is providing a breakthrough by using classification to assess recurring hacking patterns and decoding the encrypted emails of the senders. For analysis, ML-based models are trained to pinpoint any anomaly in the punctuation, email headers, body-data, and other relevant metrics. The purpose of these models is to identify whether or not an email is filled with a malicious phishing threat or not.

Traversal Detection Algorithms

Cybercriminals are increasingly keeping an eye on digital users like which websites do they use the most as well as the network of such websites. For instance, consider a restaurant business. As all of the customers order their food on the website of the restaurant, hackers exploit such websites, gain access to private customer data such as credit card details, and misuse the credentials of the visitors. This type of attack is known as a watering hole.

In these types of attacks, machine learning (ML) can be a game-changer by improving the traditional web security. For instance, it can determine if users are going to be forwarded to a dangerous website’s link through the destination path’s traversal. To attain this goal, traversal detection algorithms are integrated in ML. Likewise; ML can look for any sudden or unusual redirecting from a web-page on the host server.

Deep Learning

Ransomware is a type of cyberthreat that paralyzes and effectively locks the data of its victim. In order to provide access to this data, cybercriminals ask for ransom in exchange for data. The data is encrypted through cryptographic algorithms which generate an encryption key and sends it to the command and control center of the cybercriminals.

In such scenarios, a division of machine learning called deep learning is utilized. Deep learning is used to recognize any fresh ransomware threat. Datasets are trained for analyzing the common ransomware behaviors to predict any upcoming ransomware attack.

To make the system learn, a huge amount of ransomware files along with a bigger amount of non-malicious files are needed for training of the model. ML-based algorithms search and identify the major features from the dataset. These attributes are then subdivided in order to initiate the training of the model. Afterward, whenever a ransomware strain attempts to infect a system, the ML tool runs it against the trained model and computes a set of actions to respond to the attack, thereby saving the computer from being locked.

Remote Attacks

When a single computer or multiple computers are targeted by a cybercriminal, it is known as a remote attack. Such a hacker searches for loopholes in the network or the machine to enter a system. Usually, such attacks are carried out to copy sensitive data or completely ravage a network through a malware infection.

Remote attacks can be caused from a DDoS attack. In such types of attack, the server is damaged by repeatedly flooding it with fake requests. Consequently, as the servers are frozen, the cybercriminals make their move.

With machine learning algorithms, these attacks can be thwarted by a thorough analysis of the system behavior and pinpointing of any unusual instances which does not make sense according to the standard network activities. ML algorithms can be empowered to monitor and detect a malicious payload before it is too late.

Webshell

Webshell is a malware threat which facilitates a hacker in accessing and changing the settings of a website from the server’s web root directory. Hence, the cybercriminal has his/her hands on the complete database.

For e-commerce websites, cybercriminals can even get their hands on financial details like credit card data which can be exploited in a wide range of crimes. This is the major reason that webshell is mostly used against e-commerce websites.

By using machine learning, the figures and data of shopping carts can be analyzed and learnt by ML-based models to differ between malicious actions and standard actions. Malicious files can be fed to ML in order to enhance the training and capability of the model. This training then assists ML-based systems to pick webshells and quarantine them before they can perform harm the system.

by

What Is AIOps?


Recently I came across one of the very interesting topics- AIOps.

AIOps refers to Artificial Intelligence for IT operations. It involves the use of machine learning, big data analytics, and AI tools to automate the IT infrastructure of an organization.

In larger enterprises, the applications, systems, and services generate massive data volumes. With AIOps, organizations can utilize this data for monitoring their assets and examining their IT dependence more closely.

Capabilities

Ideally, an AIOps solution provides the following functionalities.

1- Automation for Routine Procedures

AIOps facilitates organizations to integrate automation in daily routine procedures. This automation can be performed for requests from users or to manage non-critical notifications from the system. For instance, if AIOps is used, then a help desk system can respond appropriately to a request from a user—all by itself. Similarly, AIOps tools can assess an alert from a system and evaluate if it requires any action without the need of a supervising authority.

2- Detection

AIOps can detect critical issues quicker and better than any manual strategy. When a familiar malware is detected on a non-critical system, the IT experts may try to eliminate it. In the meantime, they might miss an unusual activity or process on a critical system from a newly-arrived and sophisticated threat. As a consequence, the organization suffers a huge setback.

On the other hand, AIOps can make a difference by the use of vulnerability prioritization .i.e. it immediately notifies the authority about a possible cyber invasion for the critical system while for the non-critical system, it can respond by running an anti-malware tool.

3- Streamlining Interactions

Historically, before AIOps was in the scene, teams had to share and work on information through either meetings or exchanging data manually. AIOps can streamline the communication and coordination processes between teams and data center groups. It shows “relevant” data to all the IT groups. For this, the AIOps platform must be designed in a way so that it can monitor and analyze which type of data to present to which of the team.

Technologies

AIOps combines several techniques for aggregation, algorithms, analytics, data output, visualization, machine learning, and automation and orchestration. All of these techniques are mature and integrated.  So how do they work?

Log files, helpdesk ticketing systems, and monitoring provide data for AIOps. Then big data tools are used to properly manage and aggregate any data coming from the system as an output and convert it into a more useful format. To do this, analytics methods and procedures are used which attempt to extract raw data and transform it into a meaningful fresh piece of data. Therefore, analytics eliminates “noise” and irrelevant data. Additionally, it also searches for recurring patterns that can detect and mark common issues.

However, analytics cannot run without the use of proper algorithms. Algorithms support an AIOps solution to respond with the most appropriate course of action. Algorithms are configured to ensure that the IT staff can help the platform learn about the decisions pertaining to the application performance.

Algorithms are the center of machine learning. The AIOps platform sets out a standard for normal activities and behavior where it can continue to update by adding new algorithms with the addition of new data in the infrastructure of the organization.

Automation ensures that any AIOps tool is quick in performing the required action or set of actions. Automated tools are “forced” to act based on their communication with machine learning and analytics tools. For instance, a machine learning tool may establish that an application in a system requires additional storage to function. This piece of information is passed out to an automation tool which resolves to perform an action like adding more storage.

Lastly, to help in decision making, visualization is used in AIOps. It generates dashboards which are extremely easy to use and read. These dashboards contain graphical representations of all kinds, reports, and other visual elements to simplify different types of output. As a result, the management is able to remain in the loop and take any rational decision.

How Has AIOps Proved to Be a Breakthrough?

Before the emergence of AIOps, organizations faced difficulties because their IT personnel spent much of their time on routine and basic tasks. AIOps proved to be a breakthrough by helping organizations focus on more critical issues. As a result, such platforms have saved a great deal of time. IT personnel now attempt to train and educate AIOps platforms to become familiar with the organization’s IT infrastructure.

Afterward, it continues to update and evolve by making use of machine learning and algorithms as well as going through the “learned” history which it accumulated with the passage of time. Therefore, they provide for an excellent monitoring tool that has the “rationality” to perform many useful tasks.

Moreover, AIOps platforms examine and inspect causal relationships from various services, resources, data sources, and systems. Machine learning functionalities identify and run robust root cause analysis. As a result, troubleshooting of frequent issues is enhanced.

Furthermore, AIOps assists organizations to increase collaboration among all the departments. With the reports from visualization, team leaders are able to comprehend requirements and perform their duties with a renewed sense of direction.

The Other Side of the Coin

AIOps is extremely promising, but some analysts consider it to be unrefined. The debate that the effectiveness of an AIOps platform is as powerful as its “training” while the time needed to create, implement, and administer such a platform may be too time-consuming for many organizations.

Likewise, they argue that due to its ability to perform a wide variety of tasks, it requires trust from organizations. Since AIOps tool works autonomously, they have to be trained in such a way that they can easily adapt according to the environment of their organization and be able to accumulate and collect data, come up to the most logical conclusion, and allocate actions accordingly.

What Are Text Indexes in MongoDB?


MongoDB offers text indexes for search queries which include strings in their contents. At most, a collection in MongoDB can have no more than one text index. So the question is: how to build a text index?

Like other indexes, a text index can also be created using the db.collection.createIndex() method. Such an index can be built on a string array too. To define a text index for a field, you have to type “text” like the following instance.

db.employees.createIndex( {name:”text”} )

In this example, a text index has been built on the “name” field. Similarly, other fields can also be defined by using the same index.

Weights

While working with text indexes, you must become familiar with the concept of weight. Weight refers to an indexed field and marks its importance in comparison to other fields (which are also indexed) by processing the score for text search.

In all the indexed fields of a document, the match number is multiplied with the weight and the output is summed. MongoDB then takes the sum value and processes it to generate the document’s score.

By default, each indexed field carries a weight of 1. Weights can be modified in the db.collection.createIndex() method.

Wildcard Specifier

In MongoDB, there is also a wildcard specifier ($**). When this specifier is used in conjunction with a text index then it is referred to as a wildcard text index. What this does is that it applies indexing on every field which stores data in the form of strings for all the collection’s document. A wildcard specifier can be defined by using the following method.

db.collection.createIndex( {“S**”: “text”} )

Basically, wildcard text indexes can be seen as text indexes which work on more than a single field. To govern the query results’ ranking, weights can be specified for certain fields while building text indexes.

Case Insensitivity

The 3rd (latest) version of the text index offers support for the simple s and common c. The special T case folding found in Turkish is also supported.

 

The case insensitivity of the text index is further improved with support for diacritic insensitivity (a mark which represents different pronunciation) like É and é. This means that the text index does not differentiate between e, E, é, and É.

Tokenization

For purposes related to tokenization, the text index version 3 supports the following delimiters.

  • Hyphen
  • Dash
  • Quotation_Mark
  • White_Space
  • Terminal_Punctuation
  • Dash

For instance, if a text index finds the following text string, then it would treat spaces and “« »” as delimiters.

«Messi est l’un des plus grands footballeurs de tous les temps»

Sparse

By default, text indexes are “sparse” and therefore they do not need to be explicitly defined with the sparse option. When a document does not have a field indexed with a text index, then MongoDB does not make the document’s entry with the text index. When insertion occurs, then MongoDB does insert the document, however no addition occurs with the text index.

Limitations

Earlier, we talked about how there can be no more one text index for a collection. There are some more limitations for text indexes.

  • When a query entails the $text operator, then it is not possible to use the hint() method.
  • It is not possible for sort operations to utilize the text index’s arrangement or ordering, even if it is a compound text index.
  • It is possible to add a text index key to generate a compound index. Though, they have some limitations. For example, it is not possible to use special index types like geospatial index with a compound text index.

Moreover, while building a compound text index, all text index keys have to be defined adjacently. This specifying of index must come in the index specification document.

Lastly, if there are keys which precede the key of text index, then for executing a search with $text operation, it is necessary for the query predicate to use quality match conditions for the preceding keys.

  • For dropping a text index, it is mandatory to mention the index name in the method of db.collection.dropIndex(). In case you do not know the index name, the db.collection.getIndexes() method can be used.
  • There is no support for collation while working with text indexes. However, they do provide support for simple binary comparison.

Performance and Storage Constraints

While using text indexes, it is necessary to realize their impact on the performance and storage of your application

  • The creation of a text index is not too dissimilar to creating a huge multikey index. A text index takes considerably more time in comparison to a basic ordered index.
  • Considering the nature of applications, it is possible for text indexes to be “enormous”. They carry a single entry for an index in correspondence with every unique or special post-stemmed word for all the indexed fields whenever documents are inserted.
  • Text indexes do not save information or phrases related to a word’s proximity within the document. Therefore, queries with phrases run better in comparison if the complete collection is fitted into the RAM.
  • Text indexes have an effect on insertion operations in MongoDB. This is because MongoDB has to include entries for index with every post-stemmed string in indexed fields related to every newly-created source document.
  • While creating a big text index for a collection which exists for some time, make sure that you have a strong limit for open file descriptions.

$text Operator

The $text operator is used to conduct a textual search for a field’s contents which is text-indexed. An expression with the text operator has the following components.

  • $search – A single or multiple terms which is used by MongoDB for parsing and querying text indexes.
  • $language – An optional component which represents the language that is to be used for tokenizer, stemmer, and stop-words.
  • $caseSensitive – An optional component which is used to turn on or off the case sensitive search.
  • $diacriticSensitive – An optional component which is used to turn on or off the diacritic sensitive search.