The ubiquity of cloud-delivered applications and services and the always-on nature of personal and business communications have driven data traffic to grow at unprecedented rates and created virtualized, dynamic, and distributed application-delivery infrastructures. Assuring availability, security, and performance in such an environment poses a real challenge to IT departments. Therefore, traditional IT Ops has given way to DevOps to speed up ITs service response to rapidly-changing demands from their stakeholders. The rate of configuration changes, which include software updates in a DevOps environment, is by design an order of magnitude greater than in a traditional IT Ops environment. Now, IT organizations are trying to leverage machine learning and advanced analytics to further automate and improve responsiveness of infrastructure services. The impetus for configuration changes is originating not just from their stakeholders, but also from the increasing use of software-defined elements in the infrastructure. This new trend is referred to as Algorithmic IT Ops (AIOps). An environment that uses automated provisioning and software-defined or- chestration cannot ignore the impact of frequent configuration changes/updates (manifested in system/server logs) on application infrastructure performance. Additionally, in a distributed infrastructure, the impact of third-party-managed services on application and network performance is extremely significant. Thus, it becomes imperative to understand what rele- vant events are occuring outside the enterprises traditional infrastructure boundaries and how those events impact its ability to meet its performance objectives. Information provided by non-traditional, textualdata sources, e.g., API logs, outage updates, emails, and incident reports, that manifest outage and issues on third-party-managed infrastructures, become critical in infrastructure performance analytics. Todays performance-management tools primarily use numerical network-traffic-related data and limited textual data such as syslogs in silos. Mining pertinent information from textual log/event data, and correlating them with numerical performance data in unison on the same analytics platform will lead to much faster troubleshooting of application/service infrastructure performance issues. Considering these realities, in this Phase I SBIR project, Ennetix will develop a novel, log-driven infras- tructure analytics and management service, called LIAM, to enhance availability, security, and performance of distributed infrastructures, and greatly accelerate root-cause analysis of infratructure problems. LIAM will mine non-traditional textual data, such as system/server logs, configuration change logs, outage reports, event reports from other IT service management products, etc.; and correlate them with numerical network trace and server/host performance data. LIAM will feature advanced machine-learning techniques based on topic mining, novelty detection, and clustering; and it will be built on a scalable architecture to accom- modate other user-defined categorical data sources. LIAM will bring useful additional context to analyzing performance anomalies to reduce application/service interruptions, and accelerate root-cause identification and service restoration. The proposed solution will greatly benefit IT administrators and managers at DOE and other government organizations through a new approach for infrastructure performance management in todays cloud-based, dynamic, and distributed IT infrastructures. The wider benefits of this effort will extend well beyond the immediate DOE scientific community, and on to other enterprises, network operators, and cloud-service providers. In particular, many digital enterprises and commercial cloud-service providers can leverage the proposed service to proactively troubleshoot performance issues for their distributed application/service de- livery infrastructures.
