Blackbird | Real Time Data Collection Application For Social Media Platforms

Develop a Real-Time Analytics Platform for Social
Media Data Aggregation

Project Overview

Blackbird, an online data collection ecosystem for social media, partnered with IntellicoWorks, a provider of DevOps, SecOps, and DataOps services. IntellicoWorks offered custom DataOps and DevOps solutions to meet Blackbird’s real-time analytics needs for hardware connectors, enabling efficient data fetching, posting, and updates.

Project Requirements

The client required a platform that would keep track of;

A real-time analytics platform for hardware connectors to fetch information and post it back with versions and updates.
The platform should be able to collect data from different social media platforms in a consistent and reliable way.
The system should be able to handle large volumes of data and process it quickly and efficiently.
A solution that can monitor cron jobs in real-time and track the success and failure of requests per second.
The platform should be designed with security in mind, ensuring that user data is protected at all times.

Project Challenges

Blackbird needed a system that could re-scrape all records within < 4 hours of time windows, while each scrape record takes no more than 10 seconds.
The solution should be scalable and adaptable to changing business needs.
Blackbird expects high-quality DevOps, SecOps, and DataOps services to be provided by Intellicoworks, with a focus on delivering bespoke solutions that meet Blackbird's specific requirements.

Project Solution

Developed a series of operations, including development, cron, security, IoT, and governance, to meet Blackbird's requirements for the real-time analytics platform.
Implemented real-time monitoring for cron jobs that tracked the success and failure of requests per second. Developed a notification system that alerted the team when a cron job failed, enabling them to take immediate action and prevent any significant issues.
Created a system that allowed for re-scraping all records within < 4 hours of time windows. Optimized the scraping process to reduce the time it took to scrape each record to less than 10 seconds.
Utilized PHP Laravel, AWS Services, and SSH Shell technologies to develop a scalable and adaptable solution.
Provided high-quality DevOps, SecOps, and DataOps services that focused on delivering bespoke solutions to meet Blackbird's specific requirements.

Project Results

We worked closely with Blackbird and we got successful in delivering a real-time analytics platform for hardware connectors.
Hardware connectors can process more than a million records per day while keeping infrastructure costs to less than $1K per month.
The project has been running smoothly for over 5 years, with real-time monitoring and efficient re-scraping capabilities.
Intellicoworks' bespoke approach and use of cutting-edge technologies allowed for seamless integration with Blackbird's existing systems.
The successful implementation of the real-time monitoring and re-scraping system has enabled Blackbird to deliver accurate and timely data to its customers.

Technology Stack

"Working with Intellicoworks has been a game-changer as their expertise in DataOps and DevOps allowed us to provide accurate data aggregation and analysis within seconds. Thanks to Intellicoworks, our project has been running smoothly, with efficient real-time monitoring that ensures we deliver accurate data to our customers."

Harry ArbasaDirector of Projects, BlackBird

Intellicoworks' team of experts impressed us with their in-depth knowledge of data management and analytics. They delivered a highly scalable and efficient solution for our social media marketing platform, demonstrating their professionalism and expertise.

Steven ClaytonCEO, Netblaze

"Intellicoworks has been a great partner in developing solutions for our growing gaming platform. Their approach was scalable and sustainable, and their solutions brought benefits in trust, transparency, traceability, efficiency, speed, ownership, and enhanced security.

Jamie ThompsonFounder, VulcanForged