Salure's avatar

Error encoding model: Malformed UTF-8 characters, possibly incorrectly encoded

Hello there! I've been struggling the following error for the past two days and I would love some help on it. This is the error:

Error encoding model [App\Models\Tenants\Employee] with ID [202] to JSON: Malformed UTF-8 characters, possibly incorrectly encoded

Here's the situation; I've got a database with table full of employee data that includes a column 'achternaam' (lastname in Dutch), all of which are using utf8mb4_unicode_ci. We receive data from an external third party, parse it and put in our database. One of these employees we've imported has this surname: Özyaprak - Tüzün. Yeah, that's a lot of umlauts.

Storing this in the database is no problem and the name is shown normally in Navicat. However, when I try try to parse that entity to JSON via a simple call of return response()->json(Employee::find(202)); I get the exception above.

I'd love some ideas on what I or Laravel (e.g. via config) is doing wrong.

Edit: I should say that this error only occurs when converting to JSON. Blade has no problem displaying this data straight out of the database. No encode/decode or htmlentities required.

0 likes
7 replies
lostdreamer_nl's avatar

JSON doesnt like non UTF-8 data.

Try saving it as htmlentities before JSON encoding.


    $test = array( 'name' => 'Özyaprak - Tüzün' );
    $test['bnamela'] = htmlentities( $test['name'] );

    echo json_encode( $test );

You can also try:

return response()->json(Employee::find(202), 200, [], JSON_UNESCAPED_UNICODE); 

it just might work

Salure's avatar

Thanks @lostdreamer_nl.

I tried your first suggestion which worked. However, I'd now consider the data in my database to be faulty as the name would be stored as 'Özyaprak - Tüzün'. This would only cause more problems for the rest of the application. It doesn't seem worth it to have to decode this everywhere I use it in the project (not just Blade/Vue).

Your second suggestion I had already tried but doesn't seem to make a difference.

36864's avatar

Does calling json_encode() on that employee's name directly work?

Salure's avatar

@36864 I suppose. I tried it with the following snippet:

$employee = Employee::select('id', 'initialen', 'voornaam', 'achternaam')->find(202);
$employee->achternaam = json_encode($employee->achternaam);

return response()->json($employee);

Which gives us:

{"id":202,"initialen":"D.","voornaam":"Derya","achternaam":"\"\u00d6zyaprak - T\u00fcz\u00fcn\"","fullname":"Derya \"\u00d6zyaprak - T\u00fcz\u00fcn\""}

Vue then shows this as

"\u00d6zyaprak - T\u00fcz\u00fcn"

It of course adds a set of quotation marks due to the double json_encode (mine and response()->json())

36864's avatar
36864
Best Answer
Level 13

Could you try a couple more things for me please? Tinker would be great for rapidly testing this:

$employee = Employee::find(202);
$employee->toArray();
$employee->jsonSerialize();
$employee->toJson();
2 likes
Salure's avatar

@36864 You've put me on the right path and we've found the issue! I have an attribute which is automatically appended to an entity via the following method on the model:

public function getInitialsAttribute()
{
    return substr($this->voornaam, 0, 1) . substr($this->achternaam, 0, 1);
}

This seemed to be the evil doer. The command you had me run in Tinker all show the following value for initials:

=> [
     "id" => 202,
     "initialen" => "D.",
     "voornaam" => "Derya",
     "achternaam" => "Özyaprak - Tüzün",
     "initials" => b"DÃ",
     ...
   ]

b"DÃ"is not a very valid value =) I removed the attribute and everything worked. Now I just need to fix this little method but that's a piece of cake.

Thanks @36864!

Edit: Changing substr to mb_substr solved that issue. :)

2 likes
slothentic's avatar

Thanks for solving the hard part, I ran into the same issue when splitting up a username. Instead of using raw mb_substr I ended up using Laravel's native Str::substr() which was a drop-in replacement, and solve the issue.

Please or to participate in this conversation.