Introduction to email (pt. 2): Mail dialog / the “mail” command

The first post of the series discussed some fundamental general knowledge about email (daemons involved, protocols, etc.). It also covered building a test system for actively trying out mail-related things. (I had to update it, however, since I discovered some problems with building the VM.)

I assume that you’ve built the test system with Vagrant to follow along. If you haven’t, please refer back to the previous post to learn how to do this. You’re free to use any FreeBSD system of course. Using vagrant has a few advantages, though. The most important is that it allows you to save the state so you can continue to play with your VM while still being able to return to the clean state anytime to follow the next parts of this series. This post will demonstrate how to send mail from the console and point out a few important things involved.

Sending mail with… “mail”

Change to the appropriate directory where you keep your Vagrant file for the mail-vm. If you’ve tinkered with the VM after the first post, reset it to the last state (if it hasn’t changed, you may issue vagrant up instead of restoring the snapshot) and enter the VM:

% cd ~/vagrant/mail-vm
% vagrant snapshot restore mail1
% vagrant ssh

Thanks to the changes we made to the config in the previous post, you should be directly logged in as root. Let’s see if root has any mail. As part of FreeBSD’s base system comes the mail utility which is a very basic MUA able to compose and view messages. Execute it without any parameters for now:

# mail

No mail for root

Ok, so root does not currently have any messages. On an average FreeBSD system there’d probably be mail there as by default the system reports in for e.g. a daily security run output. But on this system there’s nothing so far. So let’s send a message now! We can also use mail for that. With the -s parameter we can specify the subject and of course we need to tell it who we’re sending the message to! When we did that, the program will let us type in the actual message. To indicate that we’re done, we need to place a single period (.) in line all by itself and hit the return key:

# mail -s "Test mail 1" root
This is a test message!
.

EOT

Mail acknowledged the action by printing EOT (end of text). Congratulations, you’ve just sent an email from root to root! And yes, this is the same kind of email that you know from writing to other people, only done locally in this case.

Hard to believe? Let’s do it again and make use of something else that you know from email: Sending a carbon copy (cc) to another user:

# mail -s "Test mail 2" root -c vagrant
This is another test message!
.

EOT

All done! But did it actually do what we wanted it to? Let’s become the vagrant user and check our mail real quick:

# su -l vagrant
% mail

>N 1 root@mail-vm.local Fri Apr 20 20:56 19/719 “Test mail 2”

Nice: There it is! Mail obviously found a message with the subject “Test mail 2” that was sent by root@mail-vm.local. Looks good so far. Let’s quit the mail utility by pressing CTRL-D or issuing the command x.

Sending and checking mail with – mail

The mail dialog

What’s next? How about sending an email message back to root – and this time tell mail to be verbose?

% mail -v -s "Test mail 3" root
And yet another!
.

EOT

This time our mail utility shows what’s usually happening in the background:

The mail dialog! This is basically what the MUA and the MTA are talking to make mail delivery happen.

Beginning of a mail dialog

Let’s take a closer look at some snippets:

[…]
root… Connecting to [127.0.0.1] via relay…
220 mail-vm ESMTP Sendmail 8.15.2/8.15.2; Fri, 20 Apr 2018 20:59:15 +0200 (CEST)
>>> EHLO mail-vm.local
250-mail-vm.local Hello localhost [127.0.0.1], pleased to meet you

[…]

Here we can see the beginning of the dialog: Our MUA (mail) connected to the MTA (Sendmail in this case), said “hello” (or rather EHLO according to the protocol rules) and was greeted by tho MTA, too. We’ll skip the next bits; client and server agree on various parameters to upgrade their connection to use encryption instead of plain text. While encryption is definitely an important topic when it comes to mail, it also makes things a fair bit more complicated and we’ll ignore it for now.

MAIL From: SIZE=48
250 2.1.0 … Sender ok
>>> RCPT To:
>>> DATA
250 2.1.5 … Recipient ok
354 Enter mail, end with “.” on a line by itself
>>> .

The MUA announces the sender and the MTA acknowledges it. Then the MUA tells the MTA the recipient as well as the actual message and the latter acknowledges it again.

250 2.0.0 w3KIxFbt000881 Message accepted for delivery
root… Sent (w3KIxFbt000881 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 mail-vm.local closing connection

Finally the MTA tells the MUA that it accepted the message and will take care of delivering it. And that concludes the mail sending action from the perspective of the MUA. The MTA has taken over and will do something with the message.

What you’ve been reading here is an example of what an SMTP dialog looks like. If the MTA figures that the message cannot be delivered locally, it will try to connect to another MTA and pass it on using the same protocol. And if it cannot deliver the message at all (e.g. the remote MTA rejected the message, probably because the recipient user does not exist), the MTA is probably configured to send a message to the original sender, letting him know that the message was lost.

Using mail to view messages

We’re done with the vagrant user for now so let’s exit back to root:

% logout

Root should have received three mails. We can use the mail command again to look at our mailbox:

# mail

The result will look like on the bottom of this picture:

Second part of the SMTP dialog & checking root’s unread mail

Right, all three were received and are there. Sendmail obviously did its job after taking over the messages from our MUA! Mail also tells us that we have three messages in total of which three are new. And it mentions /var/mail/root. What’s that? Well, it’s a file. But let’s quit the MUA again and take a closer look:

# less /var/mail/root

Messages in the inbox file

What we’ve stumbled across here is the root user’s mailbox for incoming mail (“inbox”). It’s just a file holding the text and headers of all unread messages. Alright, all the messages are there and can be accessed by all means that you typically access text files with. But what about mail? Can you use it to view the messages, too?

You bet that’s possible. Let’s run mail again:

# mail

Do you see a difference? No? Look more closely! Last time all three messages had an uppercase “N” in front of them, meaning new. Now there’s a “U”: Those messages are still unread, but they were already in the inbox last time we checked our mail.

The greater-than sign hints that mail 1 is selected. To read it, issue the command “print” (or use the abbreviation “p”). The plus character selects the next message, while minus does the opposite. If you’d like to play around with the mail MUA a little, you should know that there are many more commands like e.g. “f” to print the current message header. Should you want to know more, the manpage is your friend.

Viewing messages with mail

If we exit now, this is what mail tells us:

Saved 2 messages in mbox
Held 1 message in /var/mail/root

What does that mean?

Inbox and mbox

If you like analogies, think of /var/mail/root as the mailbox outside of your house. If you get new mail, it’ll be put into there. Let’s say you got three letters. The analogy to what we did a minute ago was going to the mailbox and take only two of the letters out to read them. After we read them we put them somewhere were we use to stash our letters as long as we think that we might need them again. The same thing happened here: There’s one message “held” in /var/mail/root because we didn’t bother to touch it, yet. The other two were moved to the “mbox”.

Ok, what’s the mbox? It’s another file that holds email messages. Actually there’s not much different about it compared to the inbox. It’s just used differently: To locally store your mail whereas your inbox is typically on a remote system. In our case both are on the same system and so it’s just removing a message from one file and putting it in another.

# head -21 ~/mbox

Here you can see the first message and the beginning of the second one (in line 21):

Contents of our mbox

Replying to and deleting messages

If we start mail again we know how to view the one remaining message. What else could we do with it? Well, we could reply to it (“r”) and delete (“d”) it then:

Replying to and deleting a message

That wasn’t too hard, was it? Asking mail for the headers returns a no applicable messages. Now root’s inbox should be empty. Let’s run mail again. What? A new message? Checking back at what we just did, it looks like we sent the reply to both vagrant and root. We didn’t mean to receive this message so let’s delete it, too.

Alright. Our inbox should now really be clean. Is it? Let’s put it to the test:

# mail

No mail for root

It is!

Emptying the inbox again

Excellent. But… How do we access the mails that we didn’t delete which were moved to the mbox? As I said before, the mbox really is only functionally different from the inbox. In fact the inbox is merely a special mbox. On our system it’s special in being the default that mail works with unless told otherwise!

Of course we can tell mail to operate on root’s mbox instead. This is done by using “-f”:

# mail -f /root/mbox

And there are our messages. No magic here.

Accessing root’s mbox

Intermission

That’s it for this article on mail. You should now have a much better understanding of what is happening when a message is being sent – and what a message actually is. Also you’ve met an old Unix tool that probably isn’t going to become your favorite MUA but still gets the job done after several decades. And while it’s not very intuitive, it just helped you to get started in better understanding email. Also it might actually still well suffice for some simple tasks. In fact we’ve only scratched the surface of the mail utility. It can do much, much more. But that’s too special a topic and way beyond the goal of this series on email.

You’ve come to the end of part two. If you’ve been following along with your Vagrant VM, stop it now, make sure that it’s powered off and create a second snapshot:

# shutdown -p now
% vagrant status
% vagrant snapshot save mail2

Until next time!

Advertisements

Introduction to email (pt. 1): Email basics

[update: Seems like the maildir patch for the Alpine mail client does not currently work with the newest version… So we need to get an older version of the ports tree – this is not ideal, but fortunately this is only a test VM where we can do bad things]
[update2: More changes… A necessary patch is no longer available from its previous location]

Email – short for electronic mail – is one of the things of modern life that we all are familiar with… Or are we? We know how to use it and probably have a rough idea of what’s going on when we press the send button. But email is actually a surprisingly complicated topic. You probably won’t notice that until you think about setting up your own mail server for the first time. Once you do however, you’re in for quite some reading.

I thought to write this mail mini series in late 2016 when I had to debug an issue with a customer’s mail system and thus figured that I really, really could use a bit more knowledge when it comes to email. I quickly went on with other tasks, but I’m still interested in the topic – and maybe I’ll find some time for it this time. This series of posts is intended as a summary of things that I found noteworthy about email.

Daemons & Terminology

When it comes to email, various daemons are involved. Daemons? Why of course! Email is native to Unix; it had precursors, yes, but it was on Unix that things really took off. That’s why for understanding email it really helps to understand Unix.

The part of the mail system that’s visible to the user is called Mail User Agent (MUA or just UA) in mail terminology. That’s what most people just call the Email client: Programs like Thunderbird, Sylpheed or Outlook. It can be used to compose messages (that’s the correct term – you’re actually sending messages not emails) and hand them over to an email server or to retrieve messages from there.

The server-side of the mail system consists of multiple daemons. First there’s the Mail Transfer Agent (MTA) that’s responsible for accepting incoming messages (e.g. from the MUA), process it and either send it to a second MTA on a foreign server or to save it locally (if the message’s recipient has an account on this server). To store a message locally, the MTA can pass it to a Mail Delivery Agent (MDA).

It’s common that an MTA accepts email that’s destined for another server and hands it over to that server’s MTA. This is called relaying. To know where it needs to send the message to, it looks at the domain part of the recipients address (xyz@example.com). Then it does a lookup for the MX (Mail eXchange) records in the DNS (Domain Name System) for that zone (DNS terminology; think domain for now.

Protocols

To enable the various daemons to interact with each other they need to follow standards in communicating. These standards come in form of so-called RFCs – if you don’t know what that is, do some quick research right now. No matter what, if it has anything to do with the Internet at all, RFCs are what define the standards that every implementation is expected to follow. There are several protocols which allow for various actions:

The MUA needs to speak SMTP, the Simple Mail Transfer Protocol, if it wants to pass a message to an MTA because that’s what this daemon is expecting. The MTA can use make use of MLTP, the Mail Local Transfer Protocol, to talk to an MDA. And if the MUA is expected to retrieve messages from the mailbox it needs to implement the POP (Post Office Protocol) or IMAP (Internet Mail Access Protocol) protocols so it can communicate with a POP or IMAP daemon which will then read messages stored on the server and send them to the MUA.

Other components

And there’s more when it comes to mail. Today’s internet is well-known as a hostile place. However when email was developed, people on the net were few and shared a common passion: Tech. Nobody meant to do the others any harm. This infantile period is far from what we experience today. But the basic principles of email haven’t changed and in fact cannot be changed easily. So the problem arises how to retroactively secure something that was designed in a carefree way and therefore proved to be inherently insecure?

When you think email today, you automatically have to think encryption, too. Otherwise there might always be someone to eavesdrop on you. When you think email you have to think authorisation. If you don’t protect your mail account somebody may break into it, steal your secrets or abuse it. You have to think spam. How do you avoid being flooded with all those useless messages that want you to buy blue pills or the like? And worse: How do you prevent spammers to use your mail server so you don’t get blacklisted? You have to think viruses, phishing, trojans, etc. Have to think security holes in your applications, newer protocol versions being established, and so on.

You can already see that mail is a rather complex topic that requires you to have a fair bit of knowledge on other topics like e.g. DNS, security and more. Fortunately all the knowledge that you need is somewhere out there. You just need some determination, a lot of free time and a bit of luck to find the relevant pieces. I cannot help you with the former two but I’ll try to provide a source that could help you learn some important things about mail without having to search on for a tutorial (even though of course I cannot cover everything).

So much for a tiny bit of theory. It’s merely meant to be enough so we can start doing something and cover more of our topic along the way.

Building a test system

So let’s build some kind of test system to play around with, shall we? Sure thing. I suggest using FreeBSD – it is an excellent choice of OS to get into the topic. No, not because I’ve come to like it quite a bit and base a lot of what I post on my blog on that system. For many things Linux would be more or less an equally good choice. When it comes to getting into mail however, it isn’t.

If you already know the whole topic quite well, you can install all the needed programs on Linux without any problems. If however you’re just starting out, FreeBSD makes the first steps so much easier as it already comes with a working mail solution by default! Setting up mail server software is a very complex and complicated thing to do and FreeBSD really makes your life so much easier in this regard by removing this obstacle for you.

In a previous tutorial on backups with Bacula I used VirtualBox together with Vagrant and it proved to be a very convenient solution. So I’m going to use it here as well. If you haven’t used VirtualBox or don’t even know what Vagrant is, I’ve written this post for you which explains things very detailed and with pictures. For creating the base box that we need, you can refer to this other post that explains everything step by step. Just make sure that you read this section before you build your base box! Because here’s the customization we need for this VM:

This post assumes version we’re using FreeBSD 11.1 – any version of 11 should be fine, though. If at the time you are reading this 12.x or even newer is out, you may want to check if fundamental things (like finally putting Sendmail to rest and importing a different MTA) have changed in between these versions. Starting with version 11.0, FreeBSD has a new installation dialog screen that let’s you choose some system hardening options. While this is a great idea and I’ve recommended to disable Sendmail in the post about building the base box for Vagrant, it would ruin the mail functionality that we’re going to use here. So make sure to not disable Sendmail for this installation!

Selecting hardening options

Ports work

Also don’t install Bacula as described in the customization section for the Vagrant base box – we’re not going to use it here. Instead do the following (if you’re not using a c-shell variant, leave out the “env” for the second command):

# svnlite co svn://svn.freebsd.org/ports/tags/RELEASE_11_0_0 /usr/ports
# env ASSUME_ALWAYS_YES=1 pkg bootstrap
# sed -i.bak -e 's|pkg$|pkg register|' /usr/share/mk/bsd.own.mk 

This downloads an old version of the ports tree which holds a version of Alpine that we’re going to use. It also bootstraps the package manager and makes a change so that it will work with the old tree. Now we need to prepare Alpine; unfortunately one required file is no longer available from the original site. I chose to mirror it since it took me some time to find the right one…

# mkdir -p /usr/ports/distfiles/alpine-2.20
# cd /usr/ports/distfiles/alpine-2.20
# fetch http://elderlinux.org/files/maildir.patch.gz
# cd /usr/ports/mail/alpine
# make makesum

Now we should be good to configure the port:

# make config-recursive

Be sure to check MAILDIR here as we’re going to need it later. Unfortunately this option is not built into Alpine would we install it as a package. So we have to resort to ports. While we’re at it, we can disable IPv6 (which we certainly don’t need), mouse support, NLS and check NOSPELL since we do not require spelling correction in this tutorial either. Configure pico-alpine accordingly. For all the other packages you can generally turn off NLS and DOCS. Then fetch the source for all packages recursively:

# make fetch-recursive

When it’s done change into OpenSMTPD’s port directory and configure it:

# cd /usr/ports/mail/opensmtpd
make config-recursive

Here you want to check the MAILERCONF option. The rest is fine. You don’t need to build the EXAMPLES for m4 and again can generally deselect NLS. Then fetch the sources:

# make fetch-recursive

Then configure dovecot2 (you can keep the default options this time) and fetch the distfiles for it:

# cd /usr/ports/mail/dovecot2
# make config fetch

Then follow the rest of the base box build process.

Preparation

You have your base box built and imported? Good. Create your Vagrant directories if you haven’t and a Vagrantfile in there:

% mkdir -p ~/vagrant/mail-vm
% cd ~/vagrant/mail-vm
% vi Vagrantfile

Put the following lines into it (assuming that your box has the name fbsd11.1-template):

Vagrant.configure("2") do |config|
config.vm.box = "fbsd11.1-template"
config.vm.network "private_network", ip: "192.168.0.10"
#config.ssh.username = "root"
config.vm.synced_folder ".", "/vagrant", disabled: true
end

Now fire up the VM, ssh into it and switch to root. Then edit rc.conf:

% vagrant up
% vagrant ssh
% sudo su -
# vi /etc/rc.conf

Change the line that defines the hostname to:

hostname="mail-vm.local"

Save the file and exit the editor. Let’s do a bad thing next: Let’s configure the VM so that we can SSH into it directly as root. If this just sent a shiver down your spine then you’re having the right feeling. You’re not supposed to do this like… ever! But now and then you feel like doing forbidden things, right? And now’s the perfect opportunity since this is just a private, non internet facing VM. Just remember to never do this on a production machine! Ok, ready? First we need to copy Vagrant’s public key and allow it for root:

# mkdir .ssh
# touch .ssh/authorized_keys
# chmod 0700 .ssh
# chmod 0600 .ssh/authorized_keys
# cp /home/vagrant/.ssh/authorized_keys /root/.ssh/authorized_keys

Now edit the SSH daemon’s config:

# vi /etc/ssh/sshd_config

Look for the line #PermitRootLogin no, remove the comment sign and change it to yes. Save and exit the editor. Then power down the VM:

# shutdown -p now

When you’re back at your host machine, edit the Vagrantfile:

% vi Vagrantfile

Add

config.ssh.username = "root"

to the configuration. Save and exit.

Test the change you just made by starting the VM and ssh’ing into it:

% vagrant up
% vagrant ssh

If everything is correct, you should have entered the VM as root now. Power down the VM again:

# shutdown -p now

Intermission

We’ve now prepared our test environment for trying out all things mail. Use vagrant to create a snapshot now to save your progress so that we can pick up right there next time. But before we do so, we need to make sure that the VM is already powered down:

% vagrant status

If it says running (virtualbox) then wait a moment and issue the same command again. It’s important that the status is poweroff (virtualbox). If you snapshot the VM while it’s powering down, your machine will always power down if you restore the previous state! As soon as it’s off, snapshot it:

% vagrant snapshot save mail1

You now have an idea of what the whole topic email is all about. And you have a test system to try out things and get familiar with mail from the very fundamentals to – perhaps – a fully working mail server. I hope that this was interesting for you and that you’ll follow the next part, too, where we’ll explore sending mail from the console and discuss using a text mode MUA.