Intervention/image package increases size for certain images

So I'm using intervention image version 2.7.2 and when I try to optimize certain images, it will increase the size instead of reducing it. Doesn't matter if I'm uploading jpg or png images.

$image = Image::make(request()->image)->save(storage_path("app/abc.jpg"), 70);

Reducing the quality parameter doesn't make much of a difference.

The weird thing is some images will get their size reduced (which is what I want) while some will have it increased!

Any idea why this behaviour?

Please note that I do not want to resize the image, I just want to reduce its quality and size.