Magento 2 - Working with Large Collections - PHP Fatal error: Allowed memory size of XXXX bytes exhausted

In Magento 2, when you try to iterate collections with 500, 3,000 or even 10,000 records that's all fine. Nonetheless the problem comes out when you have to do the same thing with larger collection as 100,000, 500,000 ou even 10,000,000 records! Then you will run "out of memory".

The error message is something like this:

PHP Fatal error: Allowed memory size of XXXX bytes exhausted (tried to allocate XXXX bytes) in vendor/magento/zendframework1/library/Zend/Db/Statement/Pdo.php on line 228

Well, there are several ways to solve it (really really many!), but Magento itself provide us a way which may be very useful to you:

\Magento\Framework\Model\ResourceModel\Iterator has a function called walk and the idea behind is: instead of reading ALL THE DATA at once, it reads row-by-row and calls the callback function. Thus the "out of memory" issue don't come out any longer, since we are now consuming small chunk's of data!

public function iterator()
{
    // you can filter by attribute
    // $this
    //    ->collection // \Magento\Catalog\Model\ResourceModel\Product\Collection
    //    ->addAttributeToFilter('name', ['like' => '%samsung%'));

    $this
        ->iterator // \Magento\Framework\Model\ResourceModel\Iterator
        ->walk(
            $this->collection->getSelect(),
            [[$this, 'callback']]
        );
}

public function callback($args)
{
    // load of the product
    $product = $this
        ->productRepository // \Magento\Catalog\Api\ProductRepositoryInterface
        ->getById($args['row']['entity_id']);

    // do whatever you want to with the $product
    print_r($product->debug());
}

Obviously it will take more time to process since it makes a full load of the product, but certainly it fits to most of the rules out there (as general background process, export/import, mass product updates) as opposed to just iterate the entire collection at once and get "out of memory" error.

I created a project on my GitHub where you can download/clone and simply run the command php bin/magento matheusgontijo:iterator and have the a module running and then adjust for your needs.

I usually consulted Inchoo's blog post (Working with large Magento collections) but for Magento 1. As I haven't found anything similar to Magento 2, I decided to create this post. Hope I helped!