Physical Security

Lowest level, if they have physical access, they can walk off and take their time rifleing through the server. Best keep it behind multiple locked doors

Operating System Hardening

Next level up, if they have full access here, they have full access to everything else (probably). The only way to really protect against this is keeping your system up to date sandboxing (Virtual Machines, Containers, jails) and segregated duties (1 server just for db, 1 server just for web, etc)

Network Hardening

If you really need security, you can airgap the machine, that is to completely remove it from the internet, but that's only practical in extreme cases where security is the utmost, or you can no longer update the system. Besides, a computer that's disconnected from the rest is just a digital spaceheater, and when its express intent is to connect to othersm you have to find the balance of letting people in, but not too much so.

Port Security

So you've got to allow people in, and in networking we come to ports. Like doorways, they can be open or closed (or just not respond when asked). Since there are 65,535 of them, the two parties have to agree which to use, and as the host, you don't want to have to answer all the doors, and thankfully, you don't have to. The principle of attack surface dictatest taht if there's only 1 point of entry to watch, then it's easier to know when something's amiss. There's a list of what ports are for which programs, so finding the right port is easier, as is knowing what the expected behavior is.

But some people advocate using non-standard ports as a security measure. The upside is for those "script kiddies" who just do drive-bys, if there's nothing readily available to see, they move on. However, this doesn't stop a persistent mal-actor from coming through and knocking on each until they get an answer. And the computer will happily do so as it can't tell from the knock if it's you, a guest, or a stranger.

There's an even more extreme method known as port knocking, that is hitting each one in a specific order adn/or timing to get the final one to open: 34,98,2,954, and finaly 22 opens. This further reduces the likelyhood of script kiddies getting in, but just as with a secret knock, a bystander could watch and make note of the order that the knocks come in, and replay the order, and once more, the computer opens.

Obscurity is not Security


So whichever way you go, you come to the next phase of security, proving you are who you say you are. There are some circumstances wherein both parties don't necessarily need to be certain of each other's identities. This page, for example, didn't ask you to log in first, i.e. didn't try to authenticate you.