1. Paper and basic spreadsheets
2. Discrete systems with little integration
3. Established software packages and basic integration
4. Hiring of business analysts
5. Building of data warehouses and data marts and beginning data governance
6. Creating a data lake
7. AI/Machine learning algorithms

1. Paper and basic spreadsheets

The very first level of business intelligence is the spreadsheet phase. This phase encompasses organizations that are generally in the very early startup phase, perhaps the first year, with fewer five staff members. In this phase the priority is often on getting the very first customers and all energy is directed to that end.

In this phase data is mostly being tracked on paper and in very basic spreadsheets. There is little data analysis going on because there’s little data to analyze. Because of the small size of the organization this fits the basic organizational profile well. Organizations generally stay at this level for less than two years – they either go out of business for lack of growth or necessity forces them to the next level.

Benefits of this stage:

• Low IT Spend
• Data is fairly accessible provided you don't lose your spreadsheets or pieces of paper

Challenges of this stage:

• No business rules on how data should be entered or validated
• Lost pieces of paper

2. Discrete systems with little integration

If an organization manages to last more than a year or two it turns its attention to the question of how to keep track of basic data in a sustainable way. This usually means replacing some of the spreadsheet and paper based methods of tracking data with some free or low-cost data systems that enforce basic business rules and provide some nice features over the spreadsheet. Typically the first systems to be implemented are basic accounting/finance packages and customer relationship management systems.

These basic systems usually offer very simple internal reporting functions and that’s generally enough for the organization to function. Organizations in this mode are in early growth mode and there’s generally not a lot of effort put into analyzing data because there is still not enough data to really change how an organization does business.

Market research may be the one exception to this rule - SEO / Pay Per Click / lead generation tools may be brought in to help a company develop its digital marketing strategy.

The core leadership team in these organizations consists of a few people who are wearing many hats so data sharing is not particularly cumbersome. If the organization grows, however, and more hires are made, it begins to segment itself into departments. That’s where the third level of data organization kicks in.

Benefits of this stage:

• Systems that enforce some level of data integrity
• Some packages have basic data analytics functions for the data stored in that system.

Challenges of this stage:

• Data silos begin to develop – no integration between packages

3. Established software packages and basic integration

As an organization grows it begins to segment into discrete departments that function somewhat independently, and these departments all start to purchase packages to manage their job functions. These packages are likely to include the following:

• Accounting/Finance package
• Customer Relationship Management Software
• Marketing Campaign Management Software
• SEO / Pay Per Click Marketing Software
• Website Content Management and Analytics Software
• Inventory Tracking
• Customer Support/Ticketing Software

Also, beyond the packages that are purchased, departments create spreadsheets and small departmental databases to support any function which is not easy in their core system. Each department in a sense becomes a mini-organization of its own and recapitulates phases 1 and 2 trying to find the right set of tools to run as a distinct department.

At this phase the organization’s founders and executives have now shifted their focus from trying to create momentum to managing the momentum that has been created. Looking at the bigger organizational picture requires gathering data from all the various departments, and this is also where the first real pains of data analytics begins to be felt.

The process of collecting data at this stage tends to follow the following template:

• Senior executives request certain data from department heads
• Department heads go into their data systems and export data into spreadsheets (if export functions are available) or manually enter data into spreadsheets
• Spreadsheets are sent to senior management for review
• Senior executives or some administrative assistants take the spreadsheets and merge them into bigger spreadsheets.
• Basic formulas and graphs and charts are created based on these spreadsheets.

Benefits of this stage:

• Departments generally have a dedicated system in place to handle their departmental data
• Lots of data is being tracked so there is enough material for analysis

Challenges of this stage:

• No “seamless” integration yet built between systems
• Cumbersome and time consuming to get all the data needed to make decisions.
• No “data specialists” who are proficient at getting data into usable formats

4. Hiring of business analysts

Once the organization grows to a certain point there is a realization that analyzing business data is time consuming enough that specialized staff should be hired. The organization then brings on board the first business analysts who are tasked with the collection and analysis of all of these spreadsheets that are flying around the organization. This frees up executives and department heads from the more arduous tasks of data analysis, but it also highlights the need for more data oriented IT infrastructure.

The newly hired business analysts are armed with a series of tools – Excel, Tableau, PowerBI, even perhaps Jupyter Notebooks. However, the realization soon sets in that all the ANALYTIC tools in the world can’t save an organization from the pain of having to have real data governance and data strategy policies that include tools and workflow to consolidate key data into one location.

This pushes organizations to the next phase – the creation of organizational data warehouses and data governance procedures.

Benefits of this stage:

• More staff to help with business intelligence
• Data visualization tools make building complex reports easier.

Challenges of this stage:

• More people in the data analysis pipeline means it can take even more time time to define and create reports
• Organization spend on data analytics goes up with the acquisition of people and tools

5. Building of data warehouses and data marts and beginning data governance

As the organization grows it begins to grapple in a serious way with questions of data governance. Some of the issues that push an organization in this direction are as follows:

• More real-time data is needed to make decisions but it is still a slow process to get data from departments.
• Data security becomes a serious issue – the organization needs to analyze sensitive information regarding employee data, healthcare data, proprietary process data – and not everyone should have access to the data. This means that policies and systems to enforce those policies have to be put into place.
• The organization may be exposed to certain regulatory frameworks (environmental, national security, etc) that require certain data to be reported on a regular basis.

The business analysts tasked with producing reports quickly begin to discover the benefit of having some sort of centralized data store that they can connect to using their reporting engines. Thus the first organizational data warehouse is born. This creates a pivot in data strategy where individual departmental data systems are looked upon more as terminals for data entry but the real analysis takes place in the data warehouse system and the surrounding toolsets.

Creating a data warehouse is no easy task and it can take years to create one. However, a good data warehouse enables decision making on a much higher level.

Benefits of this stage:

• Finally a centralized place to store and get data across the organization.
• Data warehouses bring increased redundancy and security to organizational data.

Challenges of this stage:

• Data analytics spend goes up significantly in terms of people, tools, and cloud
• Programming resources are often necessary to take advantage of this stage.

6. Creating a data lake

An organizational data lake opens up a whole new set of benefits for the organization. One of the biggest benefits of a data lake is the shift of mentality from loading THE data warehouse to loading many different data marts and data warehouses depending on the particular security and functionality needs.

Benefits of this stage:

• Finally a place to “dump” data until you decide what to do with it.

Challenges of this stage:

• Easy to go crazy on cloud spend if you aren’t careful
• Data lake can turn into “data swamp” – data is unstructured and hard to reconcile.
• Data retention policies have to be looked at and applied to data lake.

7. AI/Machine Learning algorithms

The end state of business intelligence often happens when there is so much data streaming into the organization that it becomes necessary to enlist the help of machine learning algorithms to help make sense of things. These algorithms expend tremendous processing power to extract insight from the largest of data sets.

What generally happens is that data lakes are used to store raw data in text format until a tool such as TensorFlow is deployed over the data. Cloud systems such as AWS, Azure, and Google Cloud Platform offer systems where an organization can pay for compute time to create models.

Benefits of this stage:

• Unexpected insights often come out of machine learning algorithms
• Models generated can significantly impact business and possibly allow full or partial automation of certain business functions.

Challenges of this stage:

• Significant spend on computing resources to crunch data.
• Staff with machine learning expertise are very expensive.