Hi!
You're correct in your guess there.
ServerName / ServerAlias are looking for a domain via the HTTP host header.
When your browser makes a request to your server, it sets the Host header to whatever hostname you're using in the URL. In your case, it's the IP address of the server.
So instead of Host: example.com, the web server is setting Host: your-ip-address-here.
Therefore, Apache is defaulting to your "default" virtualhost, the one that is likely found in /etc/apache2/sites-available/00-default and symlinked to /etc/apache2/sites-enabled/00-default.
Apache will default to the first virtualhost it finds when loading, which is likely 00-default (that's also why it's named 00-default - it's most likely to be the first virtual host file loaded!)
Try this:
sudo rm /etc/apache2/sites-enabled/00-default
sudo service apache2 reload
Note that I'm guessing at the file name of 00-default - I think that's correct, but I forget the exact filename.
That will disable the default site from getting loaded (note that it will not delete the configuration found in the sites-available directory, instead just deletes the symlink (alias) of that file found in the sites-enabled directory).
Then it reloads Apache's configuration.
Unless there are other websites configured in Apache, that will likely make it fall back to your new website.
Note that an alternative way to go about this would be to edit your computer's hosts file and set the hostname set in ServerName or ServerAlias to the IP address of your web server, and then use that hostname in your browser. If you restart your browser after editing your hosts file, that should load normally. However, the above steps is a more fool-proof way to get around that.
The key point here is to know how an HTTP request's Host header is important to any web server (Apache, in this case).