OpenBSD – OpenSMTPD table protocol changes, now with the backstory
https://www.undeadly.org/cgi?action=article;sid=20240503110540
Contributed by Peter N. M. Hansteen on 2024-05-03 from the tables, we deliver dept.
Regular readers will be aware that OpenBSD ships with its own mail server implementation, OpenSMTPD, in its base system.
In a recent message to the tech@ mailing list, Omar Polo (op@) asked for comments or oks for a patches implementing a change of table protocols. A little later, Gilles Chehade (gilles@) posted to the [email protected] mailing list with the backstory for this change.
The message follows in full below (apparently the otherwise fine marc.info archive site no longer archives the list):
Date: Fri, 03 May 2024 08:22:03 +0000 From: [email protected] To: [email protected] Subject: smtpd: change the table protocol Hello, This is a copy of a mail I sent to OpenBSD hackers a few days ago so you are aware of work being done on OpenSMTPD by Omar Polo. ~~~ TL;DR: proposal to change table backends wire protocol to one that's closer to filters, it has proven to work for years now, comes with many benefits and it is a very trivial change that we can pull in a handful of hours: https://tmp.omarpolo.com/smtpd-tables.7.html
Longer mail with historical context. A long time ago, OpenSMTPD shipped with two table backends for its table API: file and db. Then we started seeing a need for ldap and SQLite, so we wrote backends for them and since we had aldap.[ch] and SQLite in base this wasn't really a problem. Then we started seeing a need for mySQL and postgreSQL which became problematic because we didn't want OpenSMTPD to ship with backends that couldn't be built in base out of the box, but also because if we had to fix a bug in table_mysql, it meant we had to do an OpenSMTPD release to provide the fix for portable users. To solve this, we introduced the "procexec" API, a mean for OpenSMTPD to spawn tables in a separate process with socketpairs in place to perform lookups and obtain results via imsg. This made it possible to release table backends separately as standalone programs, without limitations to their dependencies and without requiring releases to be synchronised. Table backends were releases as part of OpenSMTPD-extras, and we could do several releases of it in the timespan of a single OpenSMTPD release. Later, we did the same for filters, it turned out to be much more complex and we failed so hard our first attempts that we reverted and waited two years to make a new proposal. That new proposal replaced imsg with an stdio line-based protocol which was much simpler for us to implement in the daemon and it had the benefit that users could easily interact with it to handle use-cases we didn't care about using the languages they wanted. I have seen some filters in awk, rust, C, golang, Perl, Python, ... users are happy with the simple API and we didn't have to handle exotic use-cases in the daemon, which is good. Recently there's been a few contributions to table-ldap in OpenSMTPD-extras, the first one in years, and I think that the fact table backends still use "procexec" is a reason why we can't rely on the community to contribute: you can't just read a line, parse it and reply, instead you need to know how imsg works, you need to understand non-blocking async I/O and you can't start a table backend of your own in a preferred language... so unless one of us does the work, it stales (I wrote table_ldap ... I don't even use or care for ldap). A while back I told eric@ we could switch tables to a filter-like protocol, he agreed, but we never got to do it. Recently, as op@ began contributing more and more, I told him about that idea again and my 3-steps plan to switch to it: 1- implement a table_stdio backend which translates imsg protocol to stdio protocol 2- convert all table backends to the new protocol 3- move the table_stdio code in smtpd so we no longer need table_stdio These are all trivial changes and op@ actually already implemented 1- and 2-, we'd like to start the work on 3- considering that all of this project is easy: we're changing the wire protocol but the table API remains 100% the same from OpenSMTPD and filters perspective. The new table protocol is identical to the smtpd-filters(7) protocol (draft by op@), it is readable in HTML at the following link and I've included the man page at end of mail: https://tmp.omarpolo.com/smtpd-tables.7.html The operations are exactly the same as with imsg, and since the imsg protocol already made results available as serialised strings, we already parse them from string to structs, and we'll re-use the same code. The change is mainly in not encapsulating them in imsg packets. ~~~ Note that we're past this as of today: 1- table_stdio was implemented as a PoC: - https://github.com/OpenSMTPD/table-stdio 2- all table backends from OpenSMTPD-extras were converted to the new interface: - https://github.com/OpenSMTPD/table-socketmap - https://github.com/OpenSMTPD/table-passwd - https://github.com/OpenSMTPD/table-ldap - https://github.com/OpenSMTPD/table-sqlite - https://github.com/OpenSMTPD/table-redis - https://github.com/OpenSMTPD/table-postgres - https://github.com/OpenSMTPD/table-mysql 3- Omar has sent a mail to tech@ with the diff to integrate protocol in smtpd: - https://marc.info/?l=openbsd-tech&m=171439783225288&w=2 There you go, now you know !