Ippon USA worked with a Fortune 500 financial services company to build a system that allows consumers to save considerable amounts of money. Our consultants leveraged open source technologies (Apache Spark, Apache Kafka) to build a Big Data platform that runs in a public cloud without interruptions of service.
The client – a top 10 US bank – wanted to allow credit card users to make better use of their money. Through the analysis of billions of credit card transactions, Data Scientists came up with multiple patterns of transactions where customers could save money. Thanks to notifications sent to their smartphone, customers can take action to claim money wrongly collected by merchants or to reduce the usage of some services.
Two main factors drove the choice of the technologies:
The data processing could be done either in batch – once a day – or in real-time – within seconds – depending on the pattern to detect. Data Scientists and Software Engineers had to collaborate even though they did not share the same programming skills: Data Scientists were familiar with Python whereas Software Engineers were more familiar with Java. Apache Spark was chosen as the processing framework and Apache Kafka as the messaging system.
Ippon USA helped set up the architecture with multiple Spark jobs: batch jobs as well as streaming jobs (using Spark Streaming), some jobs being written in Python, others being written in Scala.
Our consultants brought their knowledge and experience to build a robust platform that could handle millions of transactions per day and absorb spikes of usage. Emphasis was made on the resilience of the jobs to make sure no transaction would be lost in case of hardware or software failure.
The application runs 24/7 on a cluster in the Cloud that can be scaled up or down to adjust to the usage. The cumulated amount of money saved by clients every month exceeds one million dollars.
Collaboration was key in this project: Data Scientists were able to deliver the business logic while Software Engineers built the platform to support the execution of the system.
Ippon USA is proud to have leveraged the skills of its consultants to secure the implementation of such a critical application.