Bright Data relies on DigitalOcean’s stability to meet high-bandwidth needs

Bright Data is an industry-leading web data platform that allows businesses worldwide to collect, index, and structure public web data quickly and at scale The team at Bright Data strives to make data accessible, helping people to quickly and easily access the information they need to make the necessary strategic and operational decisions. Organizations of all sizes—from Fortune500 companies to academic institutions and small businesses—rely on Bright Data to retrieve crucial public web data in the most efficient, reliable, and flexible way so that they can research, monitor, analyze data, and make better decisions.

Founded in 2014, Bright Data was established to address the growing need for web data. The internet may be the largest database, but it is heavily segmented. For example, what is accessible in one region may not be accessible in another. To address this challenge, Bright Data began offering a residential IP network, enabling related web data collection requests to be seen and treated as requests from real consumers/users, an unprecedented approach at the time which helped businesses of all sizes access any public web data anywhere in the world… Bright Data made the decision to partner with DigitalOcean during its foundational year and has used DigitalOcean as part of its cloud infrastructure ever since. Now, they’ve scaled up to DigitalOcean Kubernetes to support their rapid growth.

High-bandwidth stability and reliability with DigitalOcean

Bright Data maintains a multi-cloud infrastructure, allowing them to rely on the different strengths of multiple cloud providers and give more flexibility to their business. DigitalOcean’s commitment to open source and transparency enables organizations to configure their infrastructure in whatever way they think is best, with DigitalOcean as a faithful and reliable partner along the way. For Bright Data, this means that they can use DigitalOcean’s scalable, reliable Droplets and low-cost bandwidth to maintain proxy servers and quickly scale with a straightforward API.

Bright Data uses DigitalOcean Droplets to deploy their web data collection platform that crawls the public internet, gathering and cataloging information. DigitalOcean’s Droplets offer a level of stability and bandwidth capabilities at a price that is lower than other Cloud Service Providers, enabling Bright Data to increase its volume and continue to scale as they grow its customer base. Scaling is made even easier through the ability to quickly deploy Droplets using DigitalOcean’s API.

“We soon found out that the service we received from DigitalOcean was the best in terms of stability of the Droplets and the bandwidth, and all of the network that comes out of the data centers. It’s very stable. This is the key point, and that stability has helped us to increase our volume as well.” - Nir Borenshtein, COO

By 2016, Bright Data maintained 10 million Residential IPs, growing to 35 million by 2018. By 2019, the internal organization had grown from 8 individuals (in 2014) to 120 team members and had launched several new product offerings. Bright Data has seen exponential growth in the last few years, now serving more than 15,000 customers, with their internal team growing to over 450 individuals spread across 17 countries. The organization has launched multiple product lines and has established main offices in the US. In 2021, they exceeded $100 million in annual revenue and acquired 3 new companies. DigitalOcean has been alongside them every step of the way.

Bright Data processes hundreds of terabytes of data through DigitalOcean servers. In early 2022, the organization ran nearly 6,000 Droplets on DigitalOcean, with a small team maintaining the entire cloud network. The reliability of DigitalOcean’s servers helps enable the Bright Data team to ship over 12,000 product releases per year, at a rate of 60-80 new product releases per day, continuously enhancing their web data platform.

The company has also built an extensive compliance department and has implemented industry-first procedures and guidelines to ensure the data they deliver is of the highest quality and reliability.

“The technology that DigitalOcean brought is a powerful thing that helps us to be so efficient.The APIs and the ability to deploy quite fast, to deploy and be scalable is valuable to us. We are doing that all the time.” - Nir Borenshtein, COO

Migrating to DigitalOcean Kubernetes

As a team dedicated to efficiency and innovation, they prioritize finding the best possible solution. Bright Data has periods of high utilization that coincide with working hours, then periods of lower utilization outside typical working hours and on weekends. Because of their dynamic load, they were underutilizing their Droplets —consistently using about 60% of their computing power. They needed the ability to automatically scale up to meet demand so they could efficiently utilize resources instead of reserving Droplets for periods of high traffic. In order to optimize performance, the team switched to DigitalOcean Kubernetes.

“Our load is dynamic, low on weekends and high during the week. We used the DigitalOcean API to create Droplets, put an image on them, and set up our system, but at the end of the day, that’s not as powerful as Kubernetes. We wanted to solve our underutilized Droplets fast, and the only solution that came to mind was DigitalOcean Kubernetes.” - Nir Borenshtein, COO

Bright Data containerized one of their flagship products, their data platform, adjusting it to work for Kubernetes, a technique known as lift and shift. The team had their minimal viable product (MVP) after about three weeks and was ready to serve customers soon after. Bright Data moved its servers to Kubernetes and automated provisioning with Terraform. They modified a process to build containers and built their own CI/CD tool.

“Kubernetes works faster, works better, and we didn’t have to come up with a new system. It allows us to scale and focus on development. With DigitalOcean Kubernetes, we can grow faster and save money by fully utilizing our Droplets.” - Nir Borenshtein, COO

Growing to meet the needs of customers

Growing this quickly comes with challenges. Chief Operating Officer, Nir Borenshtein, says that as they’ve grown, they’ve learned that the two most valuable assets they have are their people and their technology. Growing from a founding team of 8 a few years ago to a team of 450 now requires a constant reassessment of policies and procedures from leadership and continual commitment to finding and retaining top talent. Remaining a leader in the space, he says, requires an equal commitment to ideation and innovation. The organization now has over 2,200 filed patent claims and isn’t slowing down.

“We aren’t growing in a vacuum. We’re growing quickly because the need for web data is there. As we grow, the pressure to perform for our customers increases. As the need grows, so does our need for speed and innovation. We want to deliver quality and reliable data on time to everyone and continue to build technology that addresses our customers’ ever-growing data challenges .” - Nir Borenshtein, COO

Bright Data is and will continue to be an industry leader. Their commitment to excellence extends to all areas of their business, and their partnership with DigitalOcean helps them continue to focus efforts where they matter most.