WebPOS
How to Develop a Stock Calculation Service for Hundreds of Stores: TemaBit Fozzy Group Experience
The essence of the project was to develop a cloud service that allows for the quick calculation and application of various marketing activities, obtained from master systems individually for each user. We worked to offer a fast and reliable calculation of all promotions for guests of Fozzy Group stores. Promotional activities can be quite complex, intersecting with each other, so it is necessary to consider all factors and calculate everything quickly and efficiently.
When we started the project, few believed in its success within tight deadlines and with adherence to all functional requirements. But we made it .
We use the Infrastructure as a Code approach, as well as containers, scaling, and so on. The solution has undergone many modernizations, becoming a modern cloud-native service. Our engineers are well-versed in tools like Terraform and AWS Cloud, so we chose it as the basis to start working. Considering the project’s specifics and its criticality to the business, we immediately started thinking about security.
We used the KISS principle, which is why we chose Elastic Container Service, allowing us to scale effectively as the load increases and have a certain resilience in case of failure of some containers. This enables continuous processing.
For stateless tasks, we use spot instances that exist for a short time. This way, we reduced the cost of the solution.
The business logic of the WebPOS service is built on the basis of our cash register solution. We transferred the best practices and years of experience. During the service’s development, we carried out a large number of optimization measures, especially concerning database operations. We connected ElastiCache for caching. In the process of modernization, we transitioned from a monolithic application to a modern cloud-native containers solution. Now applications can scale only the necessary components and be economically efficient accordingly.
About Testing
For testing, we used a Guest behavior model based on the expected quantity of goods and the uniqueness of generated data. After migrating from a monolithic application to the cloud, we started testing and measuring the results during various changes. We initially tried one configuration of the AWS ECS cluster, then different load balancing for these clusters, documenting each progress.
After this stage, we moved on to the optimization phase. A significant amount of time was spent on data transfer between the client and the server. To reduce transport time, we changed the data exchange scenario, resulting in a reduction in JSON size and a significant speed boost.
We then optimized the database by consolidating multiple queries into one within a single session. We implemented Amazon ElastiCache, which provided a 67% speed increase.
The next stage involves optimizing the data synchronization module from various sources. We are currently working on making database updates dynamic rather than static, dependent on the time of the last update. We are also considering transitioning to Graviton instances.
The main reason for continuous optimization is to reduce costs and data update times, as elaborated below.
The next stage involves optimizing the data synchronization module from various sources. We are currently working on making database updates dynamic rather than static, dependent on the time of the last update. We are also considering transitioning to Graviton instances.
The main reason for continuous optimization is to reduce costs and data update times, as elaborated below.
About Timing and Implementation Stages
We started in February 2023. Initially, the system operated in a data center in Ukraine. The ambitious goal was to migrate to AWS for enhanced security against various attacks and scalability.
When there’s no need for a large workload, the system should automatically scale down where possible. Resources that are not needed should be returned to the cloud. This helps us save costs. With all services in a data center, there are capital expenditures on equipment. Migrating to Amazon converts our capital investments into leased resources. The less we lease, the less we pay. We had both functional and non-functional requirements dictated by our application’s needs.
There were three phases. First was adapting to Amazon. We considered an ECS cluster for our applications. Using Kubernetes for this project at this stage would be a significant overhead. Hence, following the KISS principle, we chose ECR. However, this doesn’t rule out a future transition to Kubernetes if needed. We only pay for virtual servers; infrastructure maintenance for the ECS cluster is free for us.
In the initial phase, we migrated our services from an on-demand structure to Amazon, began initial testing, and refined services. We made fundamental changes to our database structure.
Initially, when it was in data centers, we had a separate database storing promotions for each store. In Amazon, we consolidated to a single large database for all 400+ stores, effectively handling the load. It’s stable, and in case of emergencies, we can restore the database in about an hour.
This database is critical for our services. Our main service consists of two subservices — Updater and actions (calculates promotions and returns updated prices).
The second stage was production deployment. Initially, default figures were, let’s say, NFR was 13 RPS. We entered production, connected the business, and for the first month, collected information on the load generated by our client. We analyzed whether the performance of our clusters would be sufficient. After gathering data, we began improving our infrastructure to enhance the overall service performance. We added caching of queries and responses.
If a Guest adds new items to the check, a complete recalculation of all promotions occurs, as promotions are complex and adding a new item can change the structure and price of promotions entirely.
Key Metrics: How WebPOS Accelerated Company Operations
In the first phase, we achieved the following results:
The service response time before migrating to the cloud platform was 2-3 seconds.
After migrating to AWS, the response time decreased to 400-500 milliseconds, improving performance by over 8 times.
We planned to support 40 queries per second but designed the system to handle three times more queries without degradation.
The system is ready for potential increases in query volume in the future.
Metrics Before Optimization
Metrics after optimization: The number of queries per second increased by 2.5 times, and the 95% reduction was nearly 5 times.
We have created data dashboards and track key metrics, presenting them as SLA and SLI on graphs. Our services can scale automatically. We are not responsible for aspects related to physical servers and equipment access, as we pay only for the resources currently in use.
Previously, we had around 400-500 databases with a total volume of approximately 60 gigabytes.
I am text block. Click edit button to change this text. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.
About Preparedness for Potential Blackouts
The cloud provides users with greater availability, and for us as developers, it offers the ability to work remotely, increasing security. Since the system is hosted in the cloud, the impact of our energy system’s status on the service is minimal.
How the Project Can Scale
We are ready to implement our approach for other businesses quickly, with ready work schemes. The deployment and customization for a new client’s solution may take from a few days to several weeks, depending on the number of processes.
Using the Infrastructure as Code approach allows us to deploy infrastructure within a few hours, with database synchronization and customization remaining.
Regarding Market Analogues
We searched for analogues tailored to our specific needs in the Ukrainian market but did not find any. This is a unique project — a part of a larger point-of-sale solution.
Team Growth Throughout the Project
We started working on the project in February 2023, launched a pilot in June of the same year, and production in August.
The team included Alexey Ventsel, Yura Vlasenko, Roman Dumanskyi, Timofiy Orlov, and his mentors Yaroslav Laktionov and consultants (Oleksandr Sapozhnikov and Stanislav Kolenkin).
Throughout the project, all team members grew technically. We are proud that the project allowed our DevOps Yaroslav Laktionov to raise his competency level to mid-level with the support of Timofiy. Roman Dumanskyi became a senior.
Our team consists of professionals who value their work and are always ready for the challenges that the business presents.
About Future Plans
Major tasks until the end of the year include security, finances, and dependency improvements. We have also started implementing chaos testing.
The speed of introducing functional changes is very high. A new type of promotion can be created within a week or two, providing the business with an additional budget in millions of hryvnias. Problem-solving time has decreased thanks to the L2 team.
