Pixolantis's avatar

Save email body to database or textfile?

I would like to fetch emails from the Microsoft Graph API and save the email body (text and HTML) locally.

If an email has a lot of images inserted as base64, I could imagine reaching the limits of a text field in my MySQL database.

Is there a good way to save this to the database, or should I save the plain text body to the database and save the HTML email as a file?

0 likes
5 replies
LaryAI's avatar
Level 58

When dealing with large amounts of text data, such as email bodies that may contain base64 encoded images, you have a few options to consider. Here's a solution that balances performance with ease of access:

  1. Database Storage with TEXT or BLOB Types: MySQL provides different types of text fields, such as TEXT, MEDIUMTEXT, and LONGTEXT, which can store large amounts of data (up to 4 GB for LONGTEXT). If you expect the email bodies to be very large due to base64 images, you might opt for MEDIUMTEXT or LONGTEXT.

    CREATE TABLE emails (
        id INT AUTO_INCREMENT PRIMARY KEY,
        subject VARCHAR(255),
        body_text TEXT,
        body_html LONGTEXT
    );
    

    When you fetch an email from the Microsoft Graph API, you can insert the text and HTML content into the database:

    $insertEmail = $pdo->prepare("INSERT INTO emails (subject, body_text, body_html) VALUES (:subject, :body_text, :body_html)");
    $insertEmail->execute([
        'subject' => $emailSubject,
        'body_text' => $emailBodyText,
        'body_html' => $emailBodyHtml
    ]);
    
  2. File Storage for HTML Content: If you're concerned about database size and performance, you could store the plain text in the database and save the HTML content as files on the filesystem. This approach is particularly useful if you need to serve the HTML content as static files directly to web clients.

    $emailId = $pdo->lastInsertId(); // Assuming you've just inserted the email subject and text
    $htmlFilename = "email_{$emailId}.html";
    file_put_contents("/path/to/email/storage/{$htmlFilename}", $emailBodyHtml);
    

    You would then update your database schema to include a reference to the file:

    ALTER TABLE emails ADD COLUMN body_html_file VARCHAR(255);
    

    And update the file path in the database:

    $updateEmail = $pdo->prepare("UPDATE emails SET body_html_file = :body_html_file WHERE id = :id");
    $updateEmail->execute([
        'id' => $emailId,
        'body_html_file' => $htmlFilename
    ]);
    
  3. Hybrid Approach: You could also use a hybrid approach where you store the HTML content in the database by default, but if the content exceeds a certain size, you save it to a file instead. This gives you the flexibility of database storage for most emails, while avoiding the performance hit for very large emails.

    $maxSize = 65535; // Size limit for TEXT field in bytes
    if (strlen($emailBodyHtml) > $maxSize) {
        // Save to file as described above
    } else {
        // Save to database as described above
    }
    

In any case, you should also consider the security implications of storing email content, especially if it contains executable code or scripts. Always sanitize and validate the data before displaying it to users to prevent XSS attacks and other security vulnerabilities.

Remember to adjust file paths, table names, and field sizes according to your specific requirements and environment.

Tray2's avatar
Tray2
Best Answer
Level 74

Save it on disk, clobs are usually a bad idea.

1 like
Pixolantis's avatar

@Tray2 Thank you. After another test with a issue, definitely my decision too :D

Tray2's avatar

@Pixolantis If you decide to use clobs or blobs, make sure that they have their own table, that way it will not affect the storage and performance of the table you run your normal queries against.

Please or to participate in this conversation.