Big Data is all around us, it surrounds us, and in today’s digital age the possibilities tied to the collection and implementation of these large data sets are far-reaching and, in a sense, literally endless.
Nowadays Big Data plays a key role in nearly every industry, from music to e-commerce to health care to finance to law enforcement to ad-tech to agriculture and so on – everybody relies on Big Data.
In its more orthodox applications, Big Data is used to enhance operations, make better informed decisions, follow important market or research trends and additionally compile data on market and consumer intelligence to deliver smarter products and services for consumers in the long run.
On the other end of the spectrum, Big Data has also been responsible for the materialization of many medical breakthroughs, the emergence of state-of-the-art artificial intelligence capabilities and the improvement of mobile mapping and GPS capabilities, among countless other examples.
While the advent of Big Data began in the 1990s, the bulk of companies are still learning how to understand, leverage and harness these large data sets.
Although the benefits surrounding Big Data continually prove to be vital in the quest for a better, more cohesive world, the trend for suppressing what should be “public data” is almost as large as the trend for collecting this type of data, according to Bright Data CEO Or Lenchner.
Bright Data is an Israeli industry-leading company that develops Software as a Service (SaaS) products to effectively collect public web data, servicing around 25 different industries. Its portfolio ranges from Fortune 500 companies to leading universities from across the globe, companies in the public sector, e-commerce, finance, health care and security, among others – all of which rely on web-based data to form real-time automated decisions.
“The internet is the largest data source in the history of mankind,” said Lenchner. “Most of it is structured by public information, for example, the price of a product.
“Although this is public information that you and I, as individuals, can browse a website and see it – if you’re trying to automate that process of looking at public data for various reasons, it’s getting very hard, and in many cases even near impossible,” to decipher the information and structure it in a meaningful manner, he said.
THE TOOLS that Bright Data builds allow companies to view these large public data sets – without being blocked – by crowd-sourcing information from millions of users across the globe, breaking it down by geolocation, industry, trend, etc., and then forming them into actionable insights.
“If I am in London, for example, and you are in the States, if we share the same link for a specific product page and we click it at the same time, we will get different information,” Lenchner said. “It could be different prices of the same product, it could be different flight rates on a travel website, it could be different content if it’s a news website or different advertisements.
“And, it’s really hard to compete this way. It’s like you are half blind.”
Bright Data tools, in a sense, give their users the ability to approach a public website as if they are the actual consumer themselves, and view what individuals see after landing on a page in real-time.
To understand the scale of these large data sets, imagine all that is incorporated into deciphering one data point, such as the purchase of a pair of Apple Airpods, by one consumer.
What prices do they receive? What advertisements are they exposed to along with their purchase? What are the shipping times? How are the product reviews? And so on and so forth.
“There are probably hundreds of different data outputs on a single product page that each and every one of us as a consumer cares about without even noticing,” Lenchner explained.
From there, think of millions of these events taking place simultaneously across the world, and then imagine eventually possessing the capability to bring it all together and decipher the trends present within the data.
So, with the trend of websites trying to block this “public information” – for competitive reasons – from getting into the hands of other organizations becoming so prevalent in the world of Big Data collection, then how exactly does Bright Data go about gathering data points for its customers?
In order to sift through this data, Bright Data uses a robust infrastructure that they have developed for many years to access websites through multiple servers, community resource-sharing networks and multiple complementary partnerships.
Bright Data originally started in 2012, when it launched its widely popular HolaVPN – a peer-to-peer application that, since its inception, has had over 250 million installs.
“It was very unique because it was a peer-to-peer-based VPN,” Lenchner explained. “I install the VPN, and you install the VPN, and then we could use each other’s IP address to view content on the web.”
The implementation of the community-based technology was so popular that eventually Bright Data started receiving emails and phone calls from companies asking them to transform their application into one that helps businesses access web data with accuracy, considering it holds a real ability “to see the truth on the web.”
Alongside the development of the technology, Bright Data worked to develop an extensive compliance process and department to ensure that only approved and verified users can access Bright Data’s web data platforms – to safeguard both the network and its platforms.
From there, Bright Data began creating its SaaS platform in order to meet the demands of the industry. Today, Bright Data has over 10,000 customers serviced by over 300 employees across the world, and after being acquired by UK-based EMK Capital for $160 million in 2017, it has multiplied its revenues multiple times – a large jump from its humble beginnings.
IN LINE with its mission to make all public data actually public and accessible to the world around it, Bright Data began what it dubbed as The Bright Initiative organization, which devotes Bright Data’s capabilities and expertise to make a positive impact on the world.
At the onset of the coronavirus pandemic, Bright Data was approached by Samson Assuta Ashdod Hospital, which had researchers working on a theory and was deciding whether the theory was viable or that it should just be dropped altogether.
The theory, initially, was that there are enough smart devices in the world with a specific sensor that can check the oxygen levels within your blood, and that they could use this information to predict spikes in coronavirus infections and cases across the world by the use of these capabilities.
“Remember, in March 2020, at the beginning of the pandemic, no one knew anything about COVID[-19],” Lenchner said. “They came to us with a theory and we said ok... and within hours we got back with an answer for them.”
While today, that information could be deemed as unimportant, back then, it was funneled into medical protocols as the world just began to uncover the truths behind the coronavirus pandemic.
From there, Bright Data decided that it is their responsibility to offer their capabilities to the world-at-large as a service for social impact – without making a profit from it – and introduced The Bright Initiative.
“Data serves multiple purposes beyond only business purposes for a versatile amount of organizations,” said Keren Pakes, general manager of The Bright Initiative. “The Bright Initiative started with the outbreak of the pandemic, and as we started working to establish it, we saw that it far exceeded our expectations.”
One year later, The Bright Initiative currently offers their services to 170 organizations tackling the most pressing issues around the world – at no cost to the organizations – including 90 different leading universities across the world, as well as providing support to the UK government as it drives forward the implementation of their National Data Strategy (NDS).
The organizations sponsored by Bright Data’s capabilities have implemented the data provided by The Bright Initiative to perform diverse applications ranging from identifying sex-trafficking rings, locating children in abused homes, forwarding medical research or research on scientific studies (such as climate change) to even promoting diversity within the global workforce.
All and all, Bright Data is looking to break down the barriers of public data space, in the hopes of making it accessible to all; a goal that the company is well on its way to one day accomplishing for the world-at-large.
“Public data shouldn’t be limited to anyone,” Lenchner said. “It should be accessible, but that is not the story and that is not the case, and we are trying to solve that – at least with building our own tools,” one day at a time, he said.
This article was written in cooperation with Bright Data.