Migrating and managing massive amounts of data is never an easy task, especially for e-commerce websites. First, you need to safely transfer all the data to a new platform without losing important information. Then, you have to arrange all the processes to make a site function on a new platform better (or at least not worse) than on the old one. Third, you need to continuously monitor and manage your data, optimizing website performance and user experiences.
Sounds like a routine for any e-commerce store that has ever migrated to a new platform for whatever business or usability reason. But if we take into consideration that you have to move and manage a catalog not with 500 products but, say, 100K? 250K? 1M? The routine immediately becomes a pure challenge. Still, it is not impossible if you do everything carefully and use sophisticated products that help your e-commerce website operate flawlessly.
Today, we’re sharing a case study on how to successfully migrate and manage large e-commerce catalogs to Magento 2, improving scalability, performance, and indexation. The migration process was conducted by our partners from Interactiv4 (now a part of WAM Global), with Amasty products used to handle processes on the catalog backend and Hyvӓ as a frontend solution.
XXL Catalog Migration Goals and Objectives
To let you better understand the peculiarities of migrating large e-commerce catalogs with a heavy traffic load, let’s take a look at the projects we were working on.
Atida | Mipharma
Vertical: Parapharmacy
Catalog size: 50K+ products
Characteristics: very high traffic, complex business logic, multiple third-party systems, and integrations
Constraint: migration from Spryker to Magento in 3 months due to expiring license
GreenlandMX
Vertical: Spare parts for motorbikes
Catalog size: 250K+ products
Store views: 9 and growing (European expansion)
Characteristics: enormous catalog, multi-store structure, moderate but growing traffic
As you can see, one of them is not really large in a way people usually imagine, but has a high traffic volume, which actually imposes high loads on servers. If not managed properly, this may lead to poor performance and UX issues, making the website slow and unusable.
The second company is a typical representative of those “big fat” e-commerce catalogs, with even more complexity because of ongoing market expansion.
Despite the differences, both projects had a common goal:
Migrate and operate huge catalogs on Magento while preserving performance and delivering a fast user experience.
To support this, the frontend framework Hyvä was selected for its performance benefits, and Amasty became a strategic partner on the backend side (filters, labels, brands, etc.).
Amasty Product for Large Catalogs
Core Challenges: Migrating and Managing Large Catalogs at Scale
Typical challenges that anyone who has worked with 100K+ products usually faces are as follows, and the cases under consideration were not an exception:
Indexing speed. Big, multibrand e-commerce stores 9 (just like any other stores) need to update their products from time to time, which is a routine. In some cases, this could trigger re-indexation of the whole catalog. For a small catalog, it may not even be noticed, but a spontaneous re-indexation of a 250K-product monster may paralyze the whole site.
Slow loading times. In the case of large catalogs, the product page can load very slowly, to say nothing about the product feed – it can take even minutes, sometimes hours. Definitely, this is not the experience you’d like to have, especially when you’re planning to scale your catalogs further.
Cache issues (especially REDIS). Sometimes, the components of necessary tools start to load files from the backend, which may cause a huge consumption of REDIS bandwidth.
General performance degradation in both frontend and backend. All the above-mentioned challenges actually result in the general loss of performance, causing slow pages, lost traffic, and poor conversion rates.
Hyvä compatibility. In those days, most of the Magento 2 extensions were not Hyvä-compatible by default, as Hyvä was not so widespread. But as Hyvä was chosen as a frontend solution, Amasty had to adjust and develop our modules to make them work perfectly for our clients.
With Atida and Greenland, these problems were amplified by:
Huge attribute sets and rules (brands, labels, custom business logic)
Multiple store views and markets
Numerous third-party integrations (ERPs, PIMs, external services)
Tight deadlines and no room for repeated trial-and-error
If you think that the root of these problems is the platform and its capacity, you’re not actually right. Simply blaming the platform is never a solution when it comes to big catalogs with complex rules – no platform will cope with such loads in its out-of-the-box configuration. The solution requires a deep understanding of how Magento works at scale, and careful engineering around it.
Magento Data Migration Strategy
Before starting the actual performance optimization on Magento 2, Interactiv4 had to transfer the data to the new platform cleanly and reliably. To achieve this, a structured, multi-phase migration process was established.:
Stage 1. Analysis & Mapping
First, it was necessary to understand the source data types and relations, as well as product types and variations.
Then, the process of data mapping took the stage. We needed to map entities and attributes from the old system to Magento’s catalog structure (products, categories, attributes, attribute sets).
Stage 2. Cleaning & Pruning
A lion’s share of the data migration success depends on how thoroughly you clean the datasets you’re going to move, as working with a mess will only lead to issues in the new system. So, Interactiv4 had developed a custom robust framework that allowed for easily discarding obsolete, legacy, or duplicated data based on various conditions and criteria. Doing so helped exclude data that should live in other systems and not be moved to Magento. Besides, it was necessary to normalize formats where needed for consistent attributes and values.
Stage 3. Robust Import Pipeline
The same custom framework was capable of migrating any kind of data source – CSV files, databases, other platforms/APIs – into Magento entities, by full migration or by deltas. This ensured:
Repeatable, scriptable imports
Clear error handling and logging
Support for incremental updates during the cutover phase
Result: A bulletproof catalog migration path, reducing the risk of broken products, missing attributes, or inconsistent catalog structure once live.
Backend Performance Engineering for Big Catalogs
After the migration was done, the next battle was keeping Magento fast with 50K-250K+ products. This meant that something had to be done with indexing, performance, and product loading times.
Smarter Indexing
Initially, the problem was that updating several products from the catalog could trigger reindexation of the whole website, which, keeping in mind the catalog size, took 6+ hours. Obviously, it wasn’t an option to keep it like that, so we had to come up with a solution. Actually, here it is:
Magento’s Built-In Multi-Thread Indexer
With the Magento out-of-the-box multi-thread indexer. Eventually, this allowed us to enable multi- or parallel indexation of some of the indexers. Although this was not allowed for all the indexers, it was possible for key indexers, like catalog search or price indexer.
Amasty Product Labels Extension
Amasty had designed the Hyvä-compatible Product Labels extension that allowed reindexing only specific sets of products by category or brand label when those parameters were updated. This prevented full reindex cycles for every small business change, drastically reducing infrastructure stress and indexing times.
Result: Brand and product label indexation dropped from 6+ hours to about 15 minutes on a catalog of 250K+ products across 9 store views.
Asynchronous Catalog Updates
For stock changes and product updates, synchronous processing could block transactional traffic and slow the whole site. Here’s how the problem was solved:
Asynchronous catalog updates were implemented for big catalogs.
Careful control was established on how and when product updates trigger reindex and cache flushes.
Heavy background operations were assured not to interfere with checkout and browsing.
This pattern turned out to be a real success and became a standard integration strategy at Interactiv4, not just for huge catalogs but for all serious projects.
Caching & REDIS: Debugging the Hidden Costs
During performance analysis, abnormally high REDIS network usage had been discovered. Further investigation revealed that a Hyvä component was loading SVG files from REDIS instead of from the frontend/CDN, which caused unnecessary REDIS traffic and bandwidth consumption with every request.
To fix that, we came up with the following. Instead of loading SVG files from the backend, we made the system load them from the frontend, and then the CDN did the rest, saving half of the bandwidth just with this quick fix.
This example underlines a key lesson – you must trace and understand every layer, like caching, CDN, DB queries, and code, if you want to optimize performance for huge catalogs.
Amasty Product Evolution
Both Atida and Greenland became cornerstone projects for Amasty’s product evolution. In fact, Atida was an early beta user for several Amasty modules, such as Improved Layered Navigation and Product Labels. Many features and optimizations were designed and refined specifically for these clients, and only after the successful implementation and performance were they released publicly.
Working with these clients with massive catalogs made us form our roadmap philosophy in the following way: we start with an assumption that every client who comes to us has performance issues and needs our products to fix those issues. This is why we prioritize everything related to performance and your improvement requests to the existing code. If a new feature degrades performance, code is reworked—even if it requires breaking changes.
After that, we move to compliance and accessibility, like CSP compliance and regulatory requirements. This is a must-have nowadays, and every Amasty module is assured to be fully compliant with policies and regulations all over the world.
Only after performance and compliance are in good shape, we take on new features and products, integrating them with the existing solutions.
Besides, we’ve adopted a testing strategy recommended by Abode for marketplaces. It means emulating a growing number of orders, rules, products, and traffic load. This made our products suitable for big businesses with large catalogs, future-proofing scalability, and further development.
Migrating and Managing Large E-Commerce Catalogs: Results
For Atida | Mipharma:
Successful migration from Spryker to Magento.
20% increase in conversion rate.
Scalability for further business expansion.
For GreenlandMX:
1.5% increase in conversion rates.
30% better total cost of ownership (TCO).
60% increase in the speed of new implementations.
Across both:
Demonstrated that Magento + Hyvä + well-engineered extensions can deliver fast, scalable stores even with very large catalogs.
Showed that the total cost of ownership can actually be favorable compared with certain SaaS platforms, especially when transaction-based fees elsewhere would grow with order volume.
Key Takeaways for Migrating Large Catalogs
Don’t Blame the Platform by Default: Most performance issues stem from implementation details, not the core platform.
Design a Structured Migration Process: Analyze and map data carefully, clean up legacy junk, and use a robust, repeatable import layer.
Treat Performance as a Feature, Not a Nice-to-Have: Use multi-thread indexers where possible, implement partial and async reindexing, and avoid reindexing the entire catalog for small changes.
Monitor Details: Check database and cache usage, inspect logs and traces for hidden inefficiencies, and optimize small things like where SVGs are loaded from.
Use Powerful Extensions by Established Vendors: Proven Magento 2 market players like Amasty can help you find a solution that will evolve and scale together with your catalog, future-proofing your business.
Update Both Frontend and Backend: Hyvä or any other modern, sufficient frontend framework will only shine if the backend is equally optimized. Frontend will not act as a magic pill if the backend is messy and barely alive.
Managing Large Catalogs: Success or Mission Impossible?
Migrating and managing large catalogs is not a walk in the park, but with the right strategy, tools, and collaboration, it’s more than achievable. The key is in the details: optimizing every step from data migration to backend performance, ensuring that every layer, from caching to indexing, is finely tuned. By combining a well-structured migration process with thoughtful backend optimizations and the right extensions, you can build a robust e-commerce platform capable of handling massive product catalogs without compromising performance.
As we've seen, it's not just about getting the job done; it's about creating a solid foundation that scales seamlessly as your business grows. With the right approach and powerful extensions, you can turn what seems like an overwhelming challenge into a smooth, ongoing process that powers future expansion and keeps your customers happy.
