Make a passive income by building an Email Verification API

462
0

Everyone seems to be talking about generating extra sources of income for you by way of passive incomes. Truth is, there are many ways to create such a source of extra income, but few come close to being so practical and quick to set up assuming you know the basics of building APIs.

In this article we learn how to build an Email Verification API from scratch which you will then be able to post on API Marketplaces (stay tuned for a follow up video) and generate a passive income.

The goal of this video and post is to tech you how Email Verification works by taking a look into the SMTP protocol and create a service that automates the checking of inboxes to see if the server acknowledges they exist.

Today we’re not going to spend too much time on setting up Express with Typescript, as we will focus on the Email Verification its self. Instead we will use a Typescript + Express + Nodemon boilerplate I’ve already prepared which you can clone from Github.

Before we do that, let’s get the basics checked. It’s always a great idea to work on the same Node version so we have the same Node libraries available to us (aside from the dev and project dependencies we will install). I’m using Node version 16.

If you’re not running version 16, the easiest way to install v16 (and not run into version conflicts) is by way of NVM.

Install NVM on Windows

Install NVM on Mac (using Homebrew, if you don’t have Homebrew I’d encourage you to install that first, or else using the Linux link below to install NVM without Homebrew)

Install NVM on Linux

git clone git@github.com:MichaelGradek/typescript-express-nodemon-minimal-boilerplate.git

Or go to the Github repo page if you prefer alternative methods of cloning the repo:

https://github.com/MichaelGradek/typescript-express-nodemon-minimal-boilerplate

Setting up express with POST application/json requests

Our API will have one single endpoint to which our users will send an email address over a POST request using the JSON format. We will then validate the email and check if the inbox exists using SMTP commands.

Express is a pretty minimal framework (which I prefer), so there is a small step we need to do before we can easily accept JSON POST requests.

First, we need to install a small middleware for express called body-parser:

npm install body-parser --save

Now, we must import body parser and make our express app use it as a middleware. We need to make sure we mount it as middleware before the code for our request handler.

import bodyParser from "body-parser";

const PORT = 3000;
const app = express();
app.use(bodyParser.json())

Next, we can change our controller to accept post requests instead of get.

app.post("/", async (req: Request, res: Response, next: NextFunction) => {
 // code
}

Now let’s check out controller is parsing our POST body correctly by simply adding a console.log statement

app.post("/", async (req: Request, res: Response, next: NextFunction) => {
  console.log(req.body)

  res.json({hello: 'world'})
  next()
})

When you send requests, you should see the console logging the request POST body.

Now we can add some simple request checks to make sure the user has sent us an email address to perform the checks on:

app.post("/", async (req: Request, res: Response, next: NextFunction) => {
  console.log(req.body)

  if (!req.body) {
    res.status(400).json({error: 'Missing email'})
    return next()
  }

  if (!req.body.email) {
    res.status(400).json({error: 'Missing email'})
    return next()
  }

  res.json({hello: 'world'})
  next()
})

We’re just checking if the body exists, and if an email address exists in the body of the POST request. If they are missing, we return an error in JSON format and a 400 status code.

Getting familiar with the SMTP protocol

Now that we have the basic setup ready, let’s dive into how SMTP works as we will be using this protocol to check for emails.

SMTP (Simple Mail Transfer Protocol) is what all email services use to communicate with each other. In principle it works in a very simple way. The protocol defines a small set of commands you can use, and it’s pretty much all about opening a connection to a SMTP server and exchanging a series of commands to send an email.

We’re going to be leveraging this protocol and will send a series of commands to check if an email inbox exists.

The first thing we need to know is the host name of the email server we want to connect to to check for emails. Let’s say we want to check if the email address foobar@gmail.com exists. We know the domain name is gmail.com, but how do we extract the SMTP host name? We can look up the DNS settings for the domain name and check the host names under the MX records.

There’s an online tool which can help us visualise this:

https://mxtoolbox.com/SuperTool.aspx?action=mx%3agmail.com&run=toolpage

As you can see, gmail has a bunch of different host names. This is done on purpose for redundancy and ensuring that if one does not responds, you still have plenty of servers to go through.

If we get the first one, for example, we can use it to establish an SMTP connection. SMTP connections use the port 25 by default, so let’s open up our terminal and try connect to the server.

These commands will change depending on if you’re on Windows or Max/Linux. On Windows you will need to use the telnet command, while on Mac/Linux we’ll use nc (Netcat).

nc gmail-smtp-in.l.google.com 25

We’re telling Netcat to connect to the first host name on port 25. The output should be something like this:

220 mx.google.com ESMTP n7-20020a056000170700b002216d813b50si1231602wrc.1013 - gsmtp

This let’s us know we have established a connection and are ready to send more commands.

Here’s a list of the basic SMTP commands: https://serversmtp.com/smtp-commands/

We’re only going to be using the following ones:

  • EHLO: This is a mandatory first command where we identify ourselves
  • MAIL FROM: We’re telling the SMTP server on behalf of who are we sending the email
  • RCPT TO: We’re asking the server to deliver a message to this inbox
  • QUIT: Once we get the response from the RCPT command, we just quit and close the connection
nc gmail-smtp-in.l.google.com 25
220 mx.google.com ESMTP n7-20020a056000170700b002216d813b50si1231602wrc.1013 - gsmtp

EHLO mail.example.org
250-mx.google.com at your service, [79.154.191.213]
250-SIZE 157286400
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8

MAIL FROM:<name@example.org>
250 2.1.0 OK n7-20020a056000170700b002216d813b50si1231602wrc.1013 - gsmtp

RCPT TO:<foobar@gmail.com>
250 2.1.5 OK n7-20020a056000170700b002216d813b50si1231602wrc.1013 - gsmtp

Now it’s about coding this sequence of messages and assessing whether the inbox exists.

Resolving the SMTP Host name from the MX records

import {promises, MxRecord} from 'node:dns'

export const resolveMXRecords = async (domain: string): Promise<MxRecord[]> => {
  try {
    return await promises.resolveMx(domain)
  } catch(error) {
    return []
  }
}

Let’s write a little wrapper function around Node’s DNS built-in library. Essentially what we’re doing is using the promises API to resolve MX records for a given domain name, and extracting the array of MX host names. If the process fails or the domain name does not have any MX records, we simply return an empty list.

Now we have a list of MxRecords which we ideally have to sort as each MX records also carries information about in what order you should attempt the different host names. You should always evaluate host names with lower priority values and work your way up if the host name is not reachable.

const sortedMxRecords = mxRecords.sort((a, b) => a.priority - b.priority)

Transacting over SMTP

Now that we have a host name we can connect to, the final (and longest step) is to complete the SMTP transaction which includes sending a sequence of commands and evaluating the response messages.

export const testInboxOnSMTPServer = async (inboxAddress: string, smtpAddress: string): Promise<TTestInboxResult> => {
  return new Promise((resolve, reject) => {
    const result: TTestInboxResult = {
      connection_established: false,
      account_exists: false
    }

    const socket = net.createConnection(25, smtpAddress)
    const stages = {
      [SMTPStages.CHECK_CONNECTION_ESTABLISHED]: {
        expected_reply_code: '220'
      },
      [SMTPStages.SEND_EHLO]: {
        command: `EHLO mail.example.org\r\n`,
        expected_reply_code: '250'
      },
      [SMTPStages.SEND_MAIL_FROM]: {
        command: `MAIL FROM:<name@example.org>\r\n`,
        expected_reply_code: '250'
      },
      [SMTPStages.SEND_RECIPIENT_TO]: {
        command: `RCPT TO:<${inboxAddress}>\r\n`,
        expected_reply_code: '250'
      },
      [SMTPStages.SEND_QUIT]: {
        command: `QUIT\r\n`,
        expected_reply_code: '221'
      }
    }

    let response = ""
    let currentStageName = SMTPStages.CHECK_CONNECTION_ESTABLISHED

    socket.on("data", (data: Buffer) => {
      const reply = data.toString()
      response += reply

      console.log('<-- ' + reply)
      const currentStage = stages[currentStageName]

      switch (currentStageName) {
        case SMTPStages.CHECK_CONNECTION_ESTABLISHED: {
          if (!reply.startsWith(currentStage.expected_reply_code)) {
            socket.end()
            break
          }

          result.connection_established = true
          currentStageName = SMTPStages.SEND_EHLO
          const command = stages[currentStageName].command
          socket.write(command, () => {
            console.log('--> ' + command)
          })

          break
        }

        case SMTPStages.SEND_EHLO: {
          if (!reply.startsWith(currentStage.expected_reply_code)) {
            socket.end()
            break
          }

          currentStageName = SMTPStages.SEND_MAIL_FROM
          const command = stages[currentStageName].command
          socket.write(command, () => {
            console.log('--> ' + command)
          })

          break
        }

        case SMTPStages.SEND_MAIL_FROM: {
          if (!reply.startsWith(currentStage.expected_reply_code)) {
            socket.end()
            break
          }

          currentStageName = SMTPStages.SEND_RECIPIENT_TO
          const command = stages[currentStageName].command
          socket.write(command, () => {
            console.log('--> ' + command)
          })

          break
        }

        case SMTPStages.SEND_RECIPIENT_TO: {
          if (!reply.startsWith(currentStage.expected_reply_code)) {
            socket.end()
            break
          }

          result.account_exists = true
          currentStageName = SMTPStages.SEND_QUIT
          const command = stages[currentStageName].command
          socket.write(command, () => {
            console.log('--> ' + command)
          })

          break
        }
      }
    })

    socket.on("connect", (data: Buffer) => {
      console.log("Connected to SMTP " + smtpAddress)
      console.log(data)
    })

    socket.on("error", (error) => {
      clearInterval(timeoutTimer)
      reject(error)
    })

    socket.on("close", () => {
      clearInterval(timeoutTimer)
      resolve(result)
    })
  })
}

There’s many things going on here, so let’s assess each step by step.

First, we need to use a Promise to wrap our code inside of, as the net library does not provide us with a Promise-based API (at least not that I know of). We will execute all of the code from within the promise and resolve or reject it later on. We do this to make our life easier and use the async / await pattern.

Next we create a new connection using the net library. We indicate port 25 on the host name resolved by the MX records on the domain name.

In node, a new connection results in an open socket which you can use to communicate back and forth since the connection remains live and open over the period you want it to, until you close it. Other scenarios can occur such as the connection closing for other reasons (internet connection failure, server closes the connection, etc) but we won’t worry about those scenarios for now.

Going back to the socket, and how to use it, in Node a socket is represented by an open stream of data to which we can read and write to. In order to interact with the stream, we must listen to events on the stream such as the event when we get new data, or the event when an error occurs.

We define a sequence of steps using an object who’s keys is a Typescript enum, and the value is another object where we define the command to send and the expected response we should get if the command is successful.

Then, we implement the event listeners. The most complex one is obviously the socket.on("data", () => {}) event as this is where most of our logic goes.

In this part we evaluate which step of the sequence of commands we’re on and progress through it making sure we receive the right responses back from the server.

Upon completion, we simply resolve the promise.

Leave a Reply

Your email address will not be published. Required fields are marked *