Trends in business intelligence 2017-2018

Trends in business intelligence 2017-2018

In this article I will review from a perspective the trends in the field ofBusiness intelligence for the coming years. The review contains buzzwords from the field, an explanation regarding the existing subfields, the various professionals required to carry out a business intelligence project, as well as examples of systems / software, each of which is a world in its own right.

The article was written by the CEO of Influence Systems - Itamar Steinberg. Inflow is a company of experts in business intelligence.

So what is business intelligence:

This is about analyzing the information that exists in the organization's systems as well as the information concerning the organization that exists in external systems.
The products of business intelligence are visual means such as graphs, tables, pivots, clocks, maps and more.

The business intelligence tools enable the ability to "play" with the information such as slice and dice, drill down, drill through. In recent years, capabilities of ad hoc analysis, rule-based report scheduling, alerts, information sharing, user-based security and many other features have also been added.
There are many tools on the market and each of them has strengths and weaknesses based on the particular solution for which it was created.

The purpose of business intelligence is to help middle managers and senior management make decisions based on knowledge instead of feelings. Recently, a whole field was added whose goal is to help the organization offer customers adapted services based on statistics on Big data using machine learning.

The basis for any analysis and execution of a business intelligence project is the data itself.

The sources of information can be different and varied.
A. Internal systems such as:
ERP, CRM, call center, dedicated systems built for the organization, the company's website and more.

B. External systems such as:
Social networks, open press, competitor websites, external service providers that provide access to information (API) and more.

Who deals with business intelligence:
In recent years the field has become broad and there are several factors in the chain required to carry out the task.
From ETL people, through DBA, BI system analysts, frontend business intelligence implementers to mathematics and statistics people.

There are almost no people who are experts in the entire breadth of the field, even though an average job offer looks like this:

BI specialist needed:
• Extensive knowledge of databases of all types (relational / nosql / columnar)
• Complete control of several ETL tools
• PhD in statistics and mathematics, knowledge and experience in Python, R and machine learning
• Experience in at least one programming language (C# / python) - minimum 5 years
• Expertise in big data: setting up Hadoop / spark infrastructures on AWS
• Experience with BI tools tableau, sisense, BO, cognos
• Black belt in karate
• Extensive experience in repairing carburetors of TMC vehicles.

There are really no such people.

Examples of implementing business intelligence:

In traditional companies:
Analysis of sales, inventory, procurement, finance, central office...
Middle and senior managers can respond to trends in the organization, close to real time, on topics such as:

• Decrease in sales of a certain product (ERP)
• Increase in the amount of returns (ERP)
• Changes in the amount of complaints (CRM)
• Changes in revenue (ERP)
• Increase in the response time of representatives (switchboard)
• Carrying out competitions between the company's employees in customer satisfaction (CRM)
• Abandonment of customers (ERP).

Companies without physical products - internet / gaming / content / WEB product companies such as:

• Quantities are recorded for the product
• Amount of downloads of the software over time
• Percentage of content consumption / length of stay
• Revenues based on areas
• Analysis of affiliates (business partners)
• Risk analysis
• Web advertising analysis

IOT - Internet of things

• Analysis of water meters for city consumers
• Medical products that transmit data about the user
• Running and sports apps
• Measuring Internet usage

What does the field of business intelligence include?
Business intelligence is a broad technological field that includes a number of different subfields which together constitute a business intelligence solution. There is a lot of confusion in the definitions, because some of the sub-fields within the broad field are large enough to constitute a field in themselves.

Below are the main areas of business intelligence:

The field of information sources and storage
Until a few years ago, the subject of information storage was technologically static. The information was stored in relational databases which were used as data warehouses and the sources of the information were databases (mssql, mysql, postgres sql...), files (csv, txt), controllers (which usually output data to files).

The position responsible for this area is a database administrator (database administrator or DBA for short). His role is to keep the database in good order and in an optimal state. Installations, backups, optimization and malfunctions.

Trends in the field of storage:
The amount of accumulated information breaks records, organizations keep data even when there is no definition yet as to whether it will be needed. Storage prices have dropped significantly and in most cases the content is saved on cloud servers such as Amazon's cloud (S3) or Microsoft's cloud (Azure).

The information structure changes dramatically due to the need to store a lot of information in different geographical areas, on distributed servers and the need for flexibility in the data structure itself. For example, companies such as: Google, Facebook, Airbnb, fiverr, wordpress.com and many others that are required to provide a worldwide response in a quick response time.

Databases of various types began to appear both in the field of storage and access of operational systems (Nosql such as mongodb), distributed relational databases (such as: nuodb) and databases for analytical analysis (for business intelligence) columnar (for example: redshift, vertica, (in memory, realtime (for example: memsql) as well as databases for search analysis such as elastic's solutions and crate.io.

The information is not only stored in a tabular structured form (structured) but most of it as unstructured (unstructured) as thousands of millions of files are stored in distributed systems of servers such as Hadoop and their analysis is more complex since it is necessary to "extract" the information from them and query them despite being distributed.

Thus, when a question is asked, the system has to access tens/hundreds of servers to organize the information and return it in such a way that it can be decoded and analyzed, for this purpose many technological tools have been created that try to help in performing these operations.

When you use terms such as: Hadoop / / spark / map reduce hive / scala and more... you are actually touching on the field of big data which is a field in itself that includes dozens of tools and there is not really a clear definition of when you cross the line from a "normal" amount of data to big data.

The field of ETL - data integration (extract - transfer - load)

The field of data integration constitutes about 60% of the time required to carry out a business intelligence project. The ETL process includes:

1. מקור - Access to the information sources (databases / big data / files / web services / controllers...)
2. Change - Treatment of incoming data (design / change / arrangement / handling of errors)
3. target - Loading to the destination (data warehouse / external systems) in an orderly and scheduled manner

In the past this process was carried out in programming languages ​​or as processes in the database (stored procedure)
During the 90s, tools were developed to carry out these operations with visual and convenient means that were suitable only for large companies due to the price and the expertise required.
Tools such as: datastage, informatica, ssis

Trends in the field of ETL:
Dozens of tools have been developed at prices that are also suitable for small and medium-sized companies, as well as open source tools such as Pentaho and Talend, which do not cost money and have very good capabilities.
Some of the tools even work in the "cloud" fully such as: dell boomi / jitterbit

Niche tools were also created for dedicated purposes such as EDI (link between Ecommerce and logistics systems) as well as tools that know how to connect to common tools such as: google analytics, github, mailchimp salesforce, and many others.

These tools improve over time and as there are changes in the sources of information (see the field of databases), so the tools also add capabilities of reading from them accordingly, such as: access to API and webservices, direct access to Hadoop, vertica, redshift, Cassandra, reading directly from Json/xml and more.
Also, the ability to "play" with the flowing information improved significantly and became comfortable and visual.

This is a fascinating and demanding field, an expert in it touches a large part of the BI worlds as a data source and as a target requirement.
We at Influo Systems specialize in ETL in open source tools and especially in Pentaho.

The field of visualization:
This area is the icing on the cake. Presenting the information to the manager. When talking about a business intelligence system (BI), it is likely that the speaker means these systems. 

Until about a decade ago (don't take my word for it) a number of leading, expensive tools "played" in the market, which provided a response to large medium-sized companies.

Tools such as: cognos which was sold to IBM, business objects which was sold to SAP.
The tools made it possible to connect to the database (DWH), create cubes (which slowly come off the chapter), define the semantic layer for the end consumer, the ability to build reports and dashboards by an expert in the field (applying business intelligence).

In recent years dozens of good tools have been created for different purposes and the variety has created differentiation which I divide into 3 main types of tools:

The traditional tools:
The ones I mentioned above, joined by Yellowfin, Pentaho and others. These are tools that provide an organizational solution including reports and dashboards, security and management. Over the years, features have been added that I will review later.

Analysis tools / dashboards: in memory
These tools enable ad hoc analysis by storing the information as part of the tool, using the RAM memory and other innovative technologies.
For the purpose of the example: memory (RAM) prices have dropped to a level that a medium-sized organization can purchase a server with 256 gigabytes of RAM at a reasonable price (compared to the past).

Tools such as: tableau, qlikview, sisense. Each of them uses memory and CPU differently. The strength of these systems is in the implementation of dashboards, an attractive GUI display as well as ad hoc analysis.

Microsoft has also recently joined in, which has been trying for years to come in with a good quality tool and recently created powerBI which looks promising.

These systems have been very common in recent years, with tableau leading the market and sisense providing a very good solution for the niche of big data on inexpensive hardware.

These systems will usually be used by a small number of power users in the organization such as financiers, economists, IT personnel for company managers.

Less common as an organizational solution in a large organization, although it exists, usually for a large organization the traditional tools will be chosen. It is not impossible that a large company will have several tools to meet the need.

Reports only
These tools make it possible to write organizational reports in a relatively convenient and fast manner. such as crystal reports, pentaho open source, Microsoft web reporting services

Also on this subject, there are new tools developed by small companies that make it possible to produce reports quickly with advanced technology. (for example: rix ad hoc, dbxtra, holistic)

If we take as an example a company that employs about 100 senior managers and middle managers, then it is likely that the company will require hundreds of reports and the internal consumers will request new reports for the controllers. Operational reports can contain hundreds and sometimes thousands of lines for the purpose of performing the task and for that we will need one of the traditional tools.

On the other hand, business development departments, economists and CPAs will want a tool that allows for analysis over long periods, ad hoc, in order to identify trends and the need for a flexible tool is very important. Therefore, for them there will be an analytical tool that allows this.

Trends in the field of business intelligence systems - visualization

1. mobile - Some systems already have a solution and for those that don't, it's likely that it will come in soon. Full mobile support both as a responsive interface and as apps for Android and IOS with the understanding that people consume content at an increasing level from their mobile phones (because managers want their information now)

2. Embedded BI - A significant part of the software companies that have tools in the cloud / applications / websites do not want the user to be forced to enter another interface in order to view their relevant information.
Therefore, they developed the ability to integrate the entire BI system or parts of it into different systems.

For example: a CRM system that has an internal dashboard that is likely not specially developed, but a third-party BI product that has been embedded in the system.

The feature exists today in Yellowfin, sisense and later I think all the companies joined. This is a hot topic. Interface capabilities include active directory access, SSO, API...

A company that develops an application / software is not interested in investing in development personnel who will develop a sub-product that is not at the core of the application. It is better for them to use a third-party product that already exists and includes many more features than they can develop.

3. Cloud - The ability to use a browser to access information and display it while maintaining the security of the information as well as avoiding installing software on the user's computer. Some systems also transfer the development tools themselves to the WEB interface. (In some software, the development is on a physical computer and after the development is transferred to the cloud - publish)

4. Self service BI - The various companies understand that the IT department cannot provide in a reasonable time all the needs in companies with a lot of employees. 4 information systems people for 1200 employees for example. Therefore, the tools become accessible to the power user. Thus, a person who previously worked in Excel to analyze information gets direct access to the database that holds millions of records, but with convenient tools he is able to build reports, pivots, graphs and even export them and share them in the organization.
It should be noted that not every product that claims to be such is indeed such.

5. Presentations - story telling . When an economist works on a series of reports and dashboards to present to management, his final product will usually be translated into a power point presentation. Some of the tools, for example Yellowfin allows the creation of a presentation within the tool, while integrating slides and "live" reports that include graphics as well as sharing the presentation with system users without having to spend time designing a presentation or copy pasting the updated information every week. So, for example, if an economist created a presentation on product returns according to different parameters (customers / customer group / types of products) over 8 slides. Every time the consumer opens the presentation, the information will be relevant for now.

6. Social BI - Sharing the information, correspondence on the content of the information, adding layers of comments on the information / graphs / tables. For example, if a manager sees that there was a negative trend last week, he can mark with the mouse on the graph and ask a question addressed to the relevant manager who in turn can answer him. All this without leaving the tool.

7. Statistical BI - machine learning - The products, with the understanding that the topic of analysis with the help of statistical tools (R, python, spark) is becoming an integral part of the BI world. Add functionality such as: trends, linear regression into the report building mechanisms. So recently Yellowfin added trends functionality and sisense added an integration of R in the functionality. Companies use these features both in the analysis of their internal information (for example: behavior prediction, risk) and as an integral part of their product, for example: friend suggestion on Facebook, products you may be interested in on Amazon, people you might want to be in touch with on LinkedIn, a hotel in your area and you were looking for a similar one on booking.com and more...

8. IOT - internet of things : Recently, almost every measuring product is connected, a component that allows it to transmit / save data (logs) that can be transferred to the cloud. A few examples out of many: water meters (KRM), automatic gates, entrance security monitoring, pedometers, wearable products, medical devices and more..
Such as: CPAP devices that check the quality of sleep and respiratory arrest and transmit statistics to the company's website and then to the doctor. The possibilities here are unlimited except for the storage of the information and its analysis. This is of course classic Big data.

In conclusion:
world Business intelligence It is an interesting and broad field, it has several subfields, each of which can be a profession in itself. The field is changing at a very fast pace and topics such as Big Data, IOT, Machine learning and embedded BI are the buzzwords of the near future. There are many solutions of different types for different needs.

The author is Itamar Steinberg, CEO of Inflow Systems Ltd. A company of experts in the field ofBusiness intelligence. Itamar is a world-renowned business intelligence expert, author of courses and lecturer in the field. Email: itamar@inflow.co.il
LinkedIn: https://www.linkedin.com/in/steinbergitamar

The company's goal is to help businesses of all sizes (from startups to large companies) choose the tools that suit their specific needs through characterization, implementation and training.