You are asking about registration abuse and login protection, basically:
- How to mitigate account creation automation?
- How to mitigate user/password guessing?
Thankfully, these are mostly solved issues.
To avoid registration “bashing”, you need to disrupt the automation. This can be achieved by various approaches, from loose to strict:
- If the API is only intended to be accessed by a specific app/browser, you can filter-out all other visitors
- Rate-limit registration by session/IP
- Require passing a CAPTCHA
- Validate E-Mail address
- Require a private sensitive detail as proof of identity (e.g., phone-number, credit-card)
What’s nice about this, is that 1-3 are tasks that can be easily offloaded to a modern Web Application Firewall without any additional server-side coding or additional load.
For "Login Protection", in addition to the above-mentioned methods 1-3, there are two more that are recommended:
Two-Factor Authentication (2FA) - This basically means that in addition to his “regular” password, the user needs to enter another token of authentication. This token can be generated by TOTP (see Google Authenticator), or a code sent by the server via SMS, Email, Phone call, etc.
API Keys - This is considered safer, because API Keys are usually very long unique strings that have very good entropy (compared to user/pass combos). More about API Keys here.
Before I go, a small remark: I saw some previous comments referring to DDoS, and I can’t stress this enough – YOU CANNOT MITIGATE DDOS BY YOURSELF. But that’s for a different thread.
Good luck!