Multilayer Testing Services for Global Print-on-Demand E-commerce Platform

Testing an e-commerce platform that handles over 30 million orders per year is a potential QA nightmare. 

How do you ensure flawless performance across multiple devices and expanding product lines, all while keeping pace with galloping client-base growth?

This was the challenge brought to us by a leading US-based print-on-demand company. 

Their platform allows both large corporations and individual clients to create customized products in a few clicks. With over 10 million users globally and more than 26 million annual orders, they needed a robust testing solution that could scale with their success.

The Challenge

To successfully work on projects like this, you need to know the difference between complicated and complex problems.

Complicated problems have clear solutions — they just require more resources. Add more workers, increase the budget, or extend the timeline, and you’ll get there.

Complex problems don‘t have ready-made answers. First, you need to define the optimal result. Then, identify specific pain points and intersecting layers of challenges and find the best mix of approaches to address them.

Our client’s rapidly growing eCommerce platform wasn’t a complicated system needing more testers. It was a complex ecosystem requiring a tailored, adaptive approach.

For starters, we needed to clearly define what we would be working with.

After explorative testing and general QA and business analysis, we have identified the following layers of testing complexity in this project:

These challenges created a multi-layered testing ecosystem. Each layer demanded specific strategies, but we couldn’t treat them in isolation. 

Every test had to consider the entire stack, from basic functionality to AI-driven features. This interconnected approach ensured that improvements in one area didn’t create unforeseen issues in another.

Our Solution

We have assembled a dedicated manual and automated testing team led by a Project Manager. Our team expanded from 3 to 20+ members, reflecting the increasing scope and complexity of the project. To ensure fruitful collaboration, we have:

• Seamlessly integrated with the client’s in-house and remote development teams.
• Participated in daily stand-ups and weekly Scrum meetings, aligning with the client’s 2-week sprint schedule.

We have created and over years enhanced Quality Assurance strategy at the intersection of Manual and Automated QA, Cloud infrastructure and new AI features and solutions.

Comprehensive Manual Testing

Surely, we cover all the standard bases, all the classics of manual testing, but with new features popping up and millions of orders pushing the system to its limits, every day brought fresh challenges. Our manual testing team had to be part detective, part daredevil. Here are some of the testing types we covered to keep manual QA bar high:

• Functional. Ensuring all features worked as intended across the platform.
• Ad-hoc. Identifying unexpected issues through exploratory testing.
• Acceptance. Verifying that the software met all acceptance criteria.
• Smoke. Quickly assessing the stability of new builds.
• Regression. Ensuring new changes didn’t negatively impact existing functionality.
• Accessibility. Verifying the platform’s usability for users with disabilities.
• UI/UX. Assessing the user interface for consistency and user-friendliness.
• Cross-platform. Verifying consistent performance across iOS, Android, and web platforms.
• Confirmation. Validating that identified issues were properly resolved.
• End-to-end. Assessing the entire user journey from product selection to order completion.

But we didn’t stop there. Our manual QA team became the architects of testing, creating unified test plans that brought together both manual and automated efforts. The goal was to create a testing ecosystem that would evolve with the platform. We set up TestRail to showcase both manual and automated test results, complete with screenshots and comments when needed. 

Advanced Automated Testing

With millions of customizable products and a constant stream of new features, our testing needs grew exponentially. Manual testing alone couldn’t cover everything without slowing down releases. We needed a way to test more thoroughly, more frequently, and across more devices – all without ballooning our team size. Automation was the clear solution. Here’s how we built our automation strategy to meet these challenges:

• Developed scripts for simultaneous testing on multiple mobile devices.
• Implemented automated consolidation of test results into comprehensive reports.
• Introduced automatic video recordings of failed tests to aid in debugging low-reproducibility issues.
• Continuously expanded the scope of automated tests to cover an increasing portion of the application.
• Сreated a mobile framework from the ground up, integrating CI/CD via Jenkins and implemented a custom device farm.
• Engineered 80 tests for iOS applications with Swift and XCUITest; optimized the CI/CD pipeline, allowing early error detection.
• Cataloged a Pytest API framework with 100+ automated backend tests.
• Set up the mobile web (mobile browsers Safari/Chrome) Appium framework from scratch (with BrowserStack, and lately, transferred to LambdaTest) and achieved 180 e2e test coverage.

By ramping up our automation efforts, we dramatically cut down on regression testing time. This was a game-changer for our manual testers. Freed from repetitive tasks, they could focus on exploratory testing of new features, edge cases, and complex scenarios that machines can’t handle.

Cross-Platform Device Testing

Some clients have the latest iPhones; others use older Android tablets. We needed to ensure our client’s app worked great on all of them. It’s not just about different screen sizes — each device has its own quirks. To address this issue, we have:

• Used over 40 iOS and Android devices, including both flagship and older models.
• Ensured consistent performance across a wide range of hardware configurations.

AI and Machine Learning Integration

Our client took personalization to new heights with AI. They built a smart system that does more than just sort products. This AI helps customers design items, suggesting themes, colors, and layouts. It even recommends how to place photos and add special touches. It’s like having a personal designer built into the app. To ensure this AI-powered personalization worked and delivered real benefits, our team:

• Conducted comprehensive testing of new AI-driven personalization features.
• Provided regular feedback to the development team, contributing to iterative improvements.
• Ensured successful rollout of the AI-enhanced version to 100% of users.

Cloud Migration Support

For a platform handling millions of orders yearly, moving to the cloud is inevitable necessity. But migrating such a vast system, with its mountain of data and complex operations, is like performing heart surgery while the patient runs a marathon. Every piece had to move seamlessly, without disrupting the constant flow of orders.​​

• Assisted in the transition of servers and resources to cloud infrastructure.
• Maintained regular testing duties while supporting the migration process.

Scalability and Performance Testing

• Implemented load testing scenarios to simulate high-volume transaction periods.
• Verified system stability and responsiveness under peak usage conditions.

Continuous Improvement and Adaptation

• Regularly reviewed and updated testing strategies to align with the evolving product.
• Expanded our testing scope to cover newly acquired software products and integrations.

Results

We have built and enhanced a testing process as dynamic as the platform itself, catching issues before they could impact users and providing a clear, real-time picture of product quality. While it’s impossible to list every win from our years of work, here are the game-changers that kept the system rock-solid and user-friendly:

• Set up a CI/CD pipeline that quickly catches performance issues in new code.
• Created a system that links tests to problems, giving developers clear info to fix issues fast.
• Keeped the platform running smoothly across all devices, maintaining its edge over competitors.
• Built a smart testing setup that simulates high traffic with fewer resources, making regular performance checks affordable.
• Achieved 400 end-to-end iOS test coverage (Appium TestNG)
• Reduced build failures by 15%.
• Built a scalable testing framework the web app leveraging JUnit5 and Selenium; decreased testing costs by 20%. 
• Introduced automated testing protocols covering 80% of test cases; reduced manual QA efforts by 25%.
• Launched and approved 40+ major feature releases, resulting in a 15% increase in user engagement.

Impact

When a developer introduced code that slowed down the checkout process by almost 15%, our system flagged it within minutes. This quick detection allowed for immediate rollback, preventing any negative impact on UX or sales.

During the holidays, when traffic jumped 47%, the platform handled an avalanche of orders without any serious glitches. 

By implementing AI-driven test scenario generation, we doubled our test coverage while reducing the time required for test creation by 60%. This allowed us to keep pace with the client’s rapid product expansion without compromising thoroughness.

The smooth experience we created across mobile and web led to an over 25% boost in customer satisfaction scores.

Conclusion

We didn’t just run tests – we built a whole system that grows and changes with the client’s needs.

By mixing smart manual testing with cutting-edge automation, we kept the quality high even as the platform exploded in size and complexity. Instead of just keeping up with the change, we helped to drive innovation. 

Our work helped clients scale faster and safer, securing millions of new orders that lead to satisfied, returning customers. 

We became partners in the client’s success, helping them lead the industry in quality and reliability.