Data Scientists in Demand in Oil, Gas to Address Big Data Challenge


Oil and gas companies are looking to add data scientists to their employee ranks as they seek to understand Big Data and the potential benefits it can bring to the oil and gas industry.

The rise of Big Data, or datasets of over a terabyte in size that cannot be captured, stored and managed by conventional database software, has become a popular topic of discussion as industries seek to mine Big Data to improve profitability and efficiency.

Data scientists are in demand as the amount of data gathered and kept in oil and gas companies and other industries grows. By 2016, the Big Data industry is projected to be a $53.4-billion industry, Venture Beat reported in November 2013. The data scientist career path offers tremendous growth across a number of industries. The job postings for data scientists grew 15,000 percent between 2011 and 2012 alone, according to Venture Beat.

The amount of data worldwide has exploded, and analyzing large data sets would become a key basis of competition, underpinning new waves of productivity growth, innovation and consumer surplus, McKinsey & Company reported in May 2011.

“The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future,” McKinsey & Company reported.

Large volumes of data have been a fact of life in the oil and gas industry for decades; some view oil and gas as the “original” Big Data industry. Technical datasets, such as seismic surveys, can consist of many terabytes requiring special hardware and software to deal with them.

“We are also seeing more and more continuously collected data in terms of multiple sensors measuring a wide range of systems across oil fields,” said Ken Tubman, vice president of Geosciences and Reservoir Engineering with ConocoPhillips. “We have new, permanently installed systems to measure reservoir properties, production operations, drilling operations and other types of data. This new data presents challenges for real-time analysis and integration of information.”



Data analytics has become a hot area as oil and gas companies look to monitor drilling and production from thousands of wells, said Michael O’Connell, chief data scientist with TIBRO. The Big Data phenomenon took hold in the marketplace several years ago due to the amount of data flowing on equipment with sensors, mobile devices and the Internet. In 2009, the number of devices on the Internet stood at 7 billion; today, 15 billion devices are linked to the Internet. That number is expected to reach 20 billion devices by 2015, O’Connell said.

In the oil and gas space, O’Connell is seeing demand for data scientists, whose combination of technical skills and business acumen are needed to handle the growing volumes of data.

For many years, ConocoPhillips has had technical people who deal with very large data volumes.

“They stay up-to-speed on current advances in analysis, but always look to bring in new skills and tap into new technical specialties that can help our operations. The area of machine learning will be no different. We work with outside groups while we develop internal capabilities and explore applications. As this area demonstrates value, the demand will ramp up,” said Tubman.

The oil and gas industry faces the challenge of gleaning insight from Big Data, and fast data, which is data that flows quickly. Historically, fast data has been found in trading systems, but the growing predominance of sensors in wells  and general operations and the Internet of Things – or uniquely identifiable objects and their virtual representations in an Internet-like structure – means that fast data is becoming common in other parts of oil and gas operations, said O’Connell.

The rise of shale play activity has also resulted in the generation of more data, said Anthony Goldbloom, founder and CEO of Kaggle, which supplies models and software to the energy industry. The company, which was founded in Australia three and a half years ago but is now based in the United States, has worked with the University of Pennsylvania, the National Jet Propulsion Laboratory in California and with companies in health care, insurance and other industries.

Last year, Kaggle began working with an oil and gas major, who said that data from their shale operations was speaking to them but that they couldn’t hear it, and that new techniques to interpret and make sense of data were needed. Since then, the company has worked with other customers on shale plays.


“The use of data analysis in predicting demand allows for ramping up or slowing down of production to have a more optimal supply, that is a supply to more closely match actual current needs rather than an oversupply or shortage,” said Dr. Tim Lynch with North Quincy, Mass.-based Psychsoftpc, which makes high performance computers and builds turnkey Hadoop Clusters for Big Data analysis, Linux clusters, scientific workstations and Tesla personal supercomputers.

“Logistics analysis helps to find the best routes for supply and keep it flowing to meet demand. All of this has created demand for data scientists to work in the oil and gas field.”


To work as a data scientist, a good educational background in mathematics, combinatorics, (a branch of mathematics concerning the study of finite or countable discrete structures) computer science, statistics, quantitative research, science and other math, research and computer-related disciplines, is helpful.

“The key things here are: math, critical thinking, research and computers, so just about any background that  emphasizes those would do,” Lynch commented.

Data scientists working at ConocoPhillips would be expected to collaborate with other disciplines to provide integrated solutions for the company.

“There are certainly opportunities to develop new techniques at this early stage of application,” Tubman commented. “They would work with others who have specific subject matter expertise related to the business problem being addressed, such as drilling or production optimization.

“While they might not have the industry-specific knowledge at first, different perspectives from those not stuck in the same old paradigm are always welcome,” Tubman commented.


Tubman said ConocoPhillips sees a variety of educational backgrounds and different levels of expertise as paths to a
data scientist career.

While many data scientists have gone through the engineering or doctorate in statistics route, O’Connell knows of one scientist who majored in psychology, but whose data analysis work gave that person the background to work in data science.

Data scientists not only need to be able to code well, but also to know statistics and how to find correlations in data to mine large data sets, said Goldbloom. Computer science with a focus on machine learning and statistics with more focus on programming versus theory are critical for a data science education.

One interesting thing about data scientists is that they can move seamlessly from one industry to the next, applying the same data analysis techniques as they would at Google or at a major oil and gas company, said Goldbloom. But this scarce skill set is hard to find as the industry must compete with Google and companies such as Microsoft. The limited number of university training courses to educate future data scientists has also contributed to the scarcity of these workers.


“Companies are just starting to dip their fingers into the uses of Big Data,” said Goldbloom.

As companies do more and find out what’s possible, they want to do more with Big Data and hire the needed workers. Supply is starting to respond to the growing demand for data scientists, with courses launched within the past year at the University of California at Berkeley and Columbia University. However, it will take time for supply to catch up with demand.

The oil and gas industry faces several challenges in recruiting data scientists, including salary. Companies such as Google and Facebook are willing to pay huge salaries for exceptional data scientists, which are crucial to their businesses. Another challenge is workplace culture, Goldbloom said. Smart data scientists like working with other smart data scientists, and while strong data scientists are employed at oil and gas companies, the culture is not the same at companies like Google, where data scientists are often appreciated in different ways.


Despite the challenges, Goldbloom said he is bullish on the impact that Big Data analytics will have on the oil and gas industry, noting that the huge amounts of underutilized data makes oil and gas “the most well-suited industry” for Big Data type techniques.

“We recognize the oil and gas industry might not be known to students studying in this area now, but we believe we offer many exciting challenges and a great working experience,” Tubman noted. “We certainly feel the same way and hope others will as well. That will make our industry attractive to those looking to address difficult problems and make a large impact.”


Analytics may present the key to solving the oil and gas industry’s talent crunch, according to a December 2013 study from the Deloitte Center for Energy Solutions, “Oil and Gas Talent Management Powered by Analytics: Adopting Analytics to Effectively Manage Workforce Needs”.

“Given the ever-changing market fluctuations and demand center shifts, nimbleness is the key to navigating and managing the dynamic pace of change in the industry today – with data analytics at its core,” said Deloitte in the report.

Companies such as Halliburton, Noble Drilling Corp. and Royal Dutch Shell plc have turned to data analytics to identify, hire and retain workers.

Tubman said that Big Data analytics is helpful to ConocoPhillips’ workforce planning activities to build forecasting models and develop appropriate strategies to support talent acquisition.

“The insight gained allows us to be proactive and more successfully relate the impact of our organizational goals to talent planning and recruiting.”


Companies are beginning to market data and analytics software – often built from publicly available data on the web – that provides insight into the available workforce by geographies and skill sets. As the capability grows in this area, companies could find this useful in their recruiting and marketing strategies and in consideration for staffing new offices or projects.

“At ConocoPhillips, surveying employees to better understand their perspective is important,” Tubman commented. “We have done some large scale review of employee engagement, measuring factors such as work/life balance and recognition in our company. As a global organization with operations around the world, we must also localize these solicitations for feedback to allow for accurate interpretation so that the analytics have context. We have been very pleased with employee participation and have acted upon the findings when appropriate.”

“Our talent planning organization consistently delivers analytical information around the state of our workforce, and we’ve been able to develop better strategies based on that process.”

While there are probably implications around using Big Data to forecast hiring needs or to track worker attrition rates, Goldbloom sees hiring analytics as a medium value use of Big Data analytics. Instead, bigger opportunities exist in Big Data for evaluating acreage, optimizing field development plans, and well completion. Big Data algorithms can take into account “a lot more” than what a person can hold in their head; this information could be used to improve production recovery.

“We certainly think there are some clever ways companies use employees’ work history as to when they’re likely to change jobs,” said Goldbloom.

This can include when employees update their profiles on professional social media networks. However, what drives profitability in oil and gas is whether a company has good land, and whether it does a good job of making the most of it.

ConocoPhillips is addressing Big Data in many ways, Tubman noted, including the use of multiple sensors.


“We also have research programs underway to develop new methods to analyze and take advantage of the new measurements.”

Efficiency can be gained in production optimization and drilling operations through real-time analysis of multiple sensors.

“Longer term, drawing information out of the large datasets can help us determine the most efficient spacing for wells, optimum completion approach and contribute to optimization of many critical decisions.”

Continuous analysis of equipment can lead to improved predictive maintenance abilities, which results in lower operational downtime.

“That’s a boost for efficiency, but also safety – our number one concern,” Tubman commented.


Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.