Introducing a big data system for maintaining well data quality and integrity in a world of heterogeneous environment
Title
Introducing a big data system for maintaining well data quality and integrity in a world of heterogeneous environment
Subject
Gasoline
Big data
Gas industry
Petroleum prospecting
Information management
Public utilities
Data reduction
Data Analytics
Database systems
Data integration
Description
Oil and gas industry is a data-driven industry as it depends massively on information technology. According to internal statistics, the amount of data coming from upstream alone is doubling every two years. This data arrives through a wide variety of vendors' sources and is handled by various applications' repositories. Well data, in specific, is a key asset for such industry throughout the process lifetime from early exploration to production. In practice, companies often tend to create their own well data master repositories that are poorly synchronized between each other and with other databases. This results into well data residing in silos databases with no commonly defined standard. Often, there is little mechanism to cross-validate well data quality across various sources. Thus, maintaining high quality level of definitive versions of well data is a critical activity to any firm's data management strategy. Recently, Big Data technologies have evolved to quickly fetch and analyze large volumes of data that can substantially lead to an improved data quality at reasonable time. In this paper, a novel system is presented to preserve high level of well data quality in a heterogeneous environment. This system utilizes Apache Spark as a main framework for distributed processing and a mid-tier software as a data integration layer. Through a set of defined mapping rules, the system will compare data from multiple databases against the database that hosts the organizational verified data. It is typical for oil and gas companies to dedicate one master database containing the corporate standard well data. So, this database will be used as a source for comparison against well data residing in project repositories. Moreover, the system extends its functionality to cover well sub data types such as headers, check shots, deviation surveys, and picks. The final output is a data quality report that helps in making strategic decisions. 2017, Society of Petroleum Engineers
2181-2195
Creator
Mahfoodh, Abdulelah Bin
Ibrahim, Mohamad
Hawi, Maan
Hakami, Khalid
Publisher
SPE Kingdom of Saudi Arabia Annual Technical Symposium and Exhibition 2017, April 24, 2017 - April 27, 2017
Date
2017
Type
conferencePaper
Citation
Mahfoodh, Abdulelah Bin et al., “Introducing a big data system for maintaining well data quality and integrity in a world of heterogeneous environment,” Lamar University Midstream Center Research, accessed May 18, 2024, https://lumc.omeka.net/items/show/28962.