Standard engineering approaches to data management have failed to keep pace with the exponential increase in the volumes of data that we collect. Even amongst large technically savvy companies with significant IT budgets, some analysts estimate that only 5-10% of all data collected is ever effectively analyzed. So what chance small to medium-sized engineering companies with limited technical resources, and even more limited budgets?
The Two Traditional Approaches to Data Management
There are two traditional approaches to data management for small to medium engineering organizations, neither of which is optimum.
The first, and most common, is to store historical data as a loose collection of flat-files, and to analyze the data file-by-file using MS Excel, or a similar tool. Storing data as a collection of flat-files distributed throughout the organization means there is no centralized data management. Individuals often develop their own workflows based on standard Windows file functions, and it is difficult to enforce organization-wide standards for data security and data integrity. It is also difficult to identify potential data correlations that may exist across multiple individual files. Finally, MS Excel, whilst an excellent all-round business spreadsheet, is not especially optimized for performing engineering analysis, and it quickly becomes slow and cumbersome when working with large data files.
The second traditional approach is to store all critical engineering date in an industry-standard database such as MS Access, SQL Server or similar. This addresses the issue of centralized data management, but at a significant budget cost, and often at the expense of flexibility in data analysis. Databases are inherently IT-centric tools. They are expensive to set up, and require a significant commitment of IT resources to maintain. The formal requirement to work across the IT interface can often be a barrier to the ad-hoc collaboration between engineers that is a critical part of the DNA of many small and medium-sized engineering organizations. And as with MS Excel, databases are primarily business tools that are not especially optimized for performing engineering analysis.