Péter Ertner
Feb 14, 2023
In this article, we explore the primary task of business logging and data collection, as well as how to choose solutions that can be implemented without modifying existing systems. We also discuss the importance of assessing metrics, creating an overview graphic interface, post-analysis alerts and reports, and using machine learning for trend analysis and anomaly detection. Learn how to turn data wealth into real-time decisions and improve your organization's performance.
The primary task of business logging and data collection is the continuous collection and aggregation of the internal operating parameters of the organization and the creation of reports and business dashboards from the resulting data wealth, as well as the possibility of data analysis that also supports drilling.
When collecting the data, it is worth choosing solutions that can be implemented without modifying the existing systems. Accordingly, it is worth starting from the available database contents and logs files. A significant number of common frameworks (e.g. BPMN and Workflow solutions) contain built-in solutions for logging business data and metrics.
It is important to note that using business metrics differs from building a data warehouse, and its purpose is also different. In this case, we analyze business events in time series and derive real-time decisions and detect events and deviations
Assessing metrics and planning data collection
In the first phase, the metrics available in the existing systems without development that can be collected in real-time must be assessed and compared with the business needs. Methodologies and business metric recommendations are available for implementing this phase, even by business area.
The selected metrics must be made comparable. Accordingly, the standard parameters in the log of business events (time, system, administrator, area, etc.) must be defined, which each log entry must contain.
Among the collected metrics, it is worth selecting those that fundamentally determine the quality of the operation of each business area (Key Performance Indicators - KPI). Boundary conditions can be defined for these, the violation of which means an exceptional situation in a business sense. In the case of KPIs, the ideal value for the business can also be determined, which appears as a goal for the organization.
Manual analysis of collected data, creation of an overview graphic interface (dashboard).
After solving the collection of the more important and easier to connect metrics, it is worth analyzing the data together in the framework of Workshops shared with the store. After the time series analyses, it is worth creating graphic interfaces. For example, they can be created interactively on the Kibana interface of ElasticSearch, using a graphical user interface.
During the development of queries and business graphics components created and continuously tuned together with business users, the KPIs are constantly refined, and they appear more and more important for the organization. Information used in this way already represents significant business value.
During the analysis of the data, the conditions for the life cycle of the data can be determined, that is, which data should be stored and analyzed in a summarized way, while others should be stored in full depth.
Post-analysis, alerts, reports
During the continuous collection of data, the data is available over an increasingly long period of time. It is already possible to perform time series analyzes on these, which can be used to analyze trends and periodic fluctuations. In this way, it is also possible to provide business forecasts for previously defined KPIs.
Individual searches can also be performed in the collected data assets, which provide the opportunity to analyze user activity and the unique effects of business events.
If necessary, it is possible to transfer aggregated information to the data warehouse application in this phase.
Since the data has been under continuous business analysis for some time and the business limits have been fine-tuned, it is possible to set alerts. The ElasticSearch tool allows you to create automatic alerts if the data (individually or aggregated) exceeds the set thresholds. The affected business areas can even react to the alarms with deeper drill-downs and analyses, the interface makes this possible.
Queries and reports run regularly by users can be set as regular, automatically prepared reports, which are prepared by the system at a set time and sent to the set end users by email.
Machine learning
The ElasticSearch database manager has a built-in machine learning module. It can be used to analyze large amounts of time series data and automatically search for anomalies in them. This can be used to determine outliers that differ from the normal data distribution. The solution is constantly learning and can automatically recognize even long-term data fluctuations.
The exceptional cases recognized in this way can be analyzed on a graphical interface, even down to the individual data level, which data dimensions triggered them can be determined. This can be used to identify the business event or series of events responsible for the special status.
With the help of the machine learning module, forecast analyzes can also be carried out, with the help of which trends to be managed in the long term can be discovered in time.