Almost to the day, 13 years ago I was elected as a voting member of portmgr, after several years as portmgr secretary. Time has come to take a break, knowing that I leave the ports infrastructure in good hands of the current team. Over the years, many things have changed. I’m especially glad to see how easy it has become to maintain a FreeBSD system without compiling anything, with the new pkg(8) frontend and poudriere(8) backend, which is also used in the build cluster. There are still a lot of ideas for improvements, and I’m looking forward to see what the team will come up with.

For me, it’s time to focus on other areas. While you won’t see me as much in FreeBSD channels and conferences, you’ll meet me more often in DNS related fora within ICANN, CENTR, DNS-OARC, and IETF.

It’s been, and still is, a great pleasure to be part of the FreeSBD community of portmgr, committers, and contributors, and I promise to no be a stranger. Safe winds!

Baptiste Daroussin started the session with a status update on package building. All packages are now built with poudriere. The FreeBSD Foundation sponsored some large machines on which it takes around 16 hours to build a full tree. Each Wednesday at 01:00UTC the tree is snapshot and an incremental build is started for all supported released, the 2 stable branches (9 and 10) and quarterly branches for 9.x-RELEASE and 10.x-RELEASE. The catalogue is signed on a dedicated signing machine before upload. Packages can be downloaded from 4 mirrors (us-west, us-east, UK, and Russia) and feedback so far has been very positive.

He went on to note that ports people need better coordination with src people on ABI breakage. We currently only support i386 and amd64, with future plans for ARM and a MIPS variant. Distfiles are not currently mirrored (since fixed), and while it has seen no progress, it’s still a good idea to build a pkg of the ports tree itself.

pkg 1.3 will include a new solver, which will help 'pkg upgrade' understand that an old packages needs to be replaced with a newer one, with no more need for 'pkg set' and other chicanery. Cross building ports has been added to the ports tree, but is waiting for pkg-1.3. All the dangerous operations in pkg have now been sandboxed as well.

EOL for pkg_tools has been set for September 1st. An errata notice has gone out that adds a default pkg.conf and keys to all supported branches, and nagging delays have been added to ports.

Quarterly branches based on 3 month support cycle has been started on an evaluation basis. We’re still unsure about the manpower needed to maintain those. Every quarter a snapshot of the tree is created and only security fixed, build and runtime fixed, and upgrades to pkg are allowed to be committed to it. Using the MFH tag in a commit message will automatically send an approval request to portmgr and an mfh script on Tools/ makes it easy to do the merge.

Experience so far has been good, some minor issues to the insufficient testing. MFHs should only contain the above mentioned fixes; cleanups and other improvements should be done in separate commits only to HEAD. A policy needs to be written and announced about this. Do we want to automatically merge VuXML commits, or just remove VuXML from the branch and only use the one in HEAD?

A large number of new infrastructure changes have been introduces over the past few months, some of which require a huge migration of all ports. To speed these changes up, a new policy was set to allow some specific fixes to be committed without maintainer approval. Experience so far has been good, things actually are being fixed faster than before and not many maintainers have complained. There was agreement that the list of fixes allowed to be committed without explicit approval should be a specific whitelist published by portmgr, and not made too broad in scope.

Erwin Lansing quickly measured the temperature of the room on changing the default protocol for fetching distils from MASTER_SITE_BACKUP from ftp to http. Agreement all around and erwin committed the change.

Ben Kaduk gave an introduction and update on MIT’s Athena Environment with some food for thought. While currently not FreeBSD based, he would like to see it become so. Based on debian/ubuntu and rolled out on hundreds of machines, it now has it’s software split into about 150 different packages and metapackages.

Dag-Erling Smørgrav discussed changes to how dependencies are handled, especially splitting dependencies that are needed at install time (or staging time) and those needed at run time. This may break several things, but pkg-1.3 will come with better dependency tracking solving part of the problem.

Ed Maste presented the idea of “package transparency”, loosely based on Google’s Certificate Transparency. By logging certificate issuance to a log server, which can be publicly checked, domain owners can search for certificates issued for their domains, and notice when a certificate is issued without their authority. Can this model be extended to packages? Mostly useful for individually signed packages, while we currently only sign the catalogue. Can we do this with the current infrastructure?

Stacy Son gave an update on Qemu user mode, which is now working with Qemu 2.0.0. Both static and dynamic binaries are supported, though only a handful of system call are supported.

Baptiste introduced the idea of having pre-/post-install scripts be a library of services, like Casper, for common actions. This reduces the ability of maintainers to perform arbitrary actions and can be sandboxed easily. This would be a huge security improvement and could also enhance performance.

Cross building is coming along quite well and most of the tree should be able to be build by a simple 'make package'. Major blockers include perl and python.

Bryan Drewery talked about a design for a PortsCI system. The idea is that committer easily can schedule a build, be it an exp-run, reference, QAT, or other, either via a web interface or something similar to a pull request, which can fire off a build.

Steve Wills talked about using Jenkins for ports. The current system polls SVN for commits and batches several changes together for a build. It uses 8 bhyve VMs instances, but is slow. Sean Bruno commented that there are several package building clusters right now, can they be unified? Also how much hardware would be needed to speed up Jenkins? We could duse Jenkins as a fronted for the system Bryan just talked about. Also, it should be able to integrate with phabricator.

Erwin opened up the floor to talk about freebsd-version(1) once more. It was introduced as a mechanism to find out the version of user land currently running as uname -r only represents the kernel version, and would thus miss updates of the base system that do no touch the kernel. Unfortunately, freebsd-version(1) cannot really be used like this in all cases, it may work for freebsd-update, but not in general. No real solution was found this time either.

The session ended with a discussion about packaging the base system. It’s a target for FreeBSD 11, but lots of questions are still to be answered. What granularity to use? What should be packages into how many packages? How to handle options? Where do we put the metadata for this? How do upgrades work? How to replace shared libraries in multiuser mode? This part also included the quote of the day: “Our buildsystem is not a paragon of configurability, but a bunch of hacks that annoyed people the most.”

Thanks to all who participated in the working group, and thanks again to DK Hostmaster for sponsoring my trip to BSDCan this year, and see you at the Ports and Packages WG meet up at EuroBSDCon in Sofia in September.

The FreeBSD project has provided pre-built ready-to-install binary packages for many years on a best-effort basis. While these packages do work in a large number of cases, there are too many inconsistencies and failure combinations, from the unpredictable update frequency to dependency handling across upgrades, for them to be used on a wider scale. After many months of work, we’re nearing a paradigm shift in both the format of the packages, and the building and distribution of the packages with the new PKGNG tools.

At the upcoming Developer Summit at the EuroBSDCon conference in Malta on September 26 and 27, there will be another Ports and Packages Summit, which will center on a round-table brainstorm that begins with a summary of the tremendous progress made in the last 12 months, and closes with a discussion of the roadmap on how to improve binary package creation, distribution, installation and upgrading. Please contact me if you have any topics you’d like to present or discuss. It will be an informal gathering, no formal slides or presentations are required.

As always, the DevSummit is an invitation-only event, so also contact me at erwin@FreeBSD.org if you want to participate.

With only three weeks to go, we so far have 7 people registered for the Ports and Packages Summit during the DevSummit at EuroBSDCon in Warsaw.
I’m sure that can’t be right. If you intend to come, please register (by sending an email to me) as soon as possible. If you don’t intend to come, please reconsider.

So far we have 4 main topics to discuss in 2 1,5 hour slots:
– Status of the move to subversion
– Status of the implementation and uptake of the new package tools
– Status and proposed schedule for scheduled releases of binary packages
– Quality assurance in all shapes and forms: QAT, redports, pointyhat

Please send any topics you’d like to discuss, presentations to present, and other items that should go on the schedule to me in the next week or two so I can prepare a draft agenda at least a week before.

Thank you and see you there!

Just a quick note to point to my slides that summarize the Ports and Packages Summit at the FreeBSD DevSummit during BSDCan 2011, which can be found here. Also, we looking forward to feedback on the PKGNG project that was announced earlier and will replace the current pkg_* tools to handle ports installation and package handling and which will be a focus for portmgr over the next few months.

On behalf of portmgr, I am pleased to announce that portmgr has found a new secretary: Thomas Abthorpe. Thomas has been a FreeBSD ports committer since 2007 and has made more than 1000 commits since. He has previously served on the ports-security team and is currently a member of the KDE and donation teams. He has also mentored several new ports committers over the years.

In his role as portmgr secretary, Thomas will help portmgr keep track of ongoing issues, keeps the portmgr, and other bookkeeping work like organizing votes and stay in touch with other FreeBSD teams.

Please welcome him onboard!

The ports tree is now tagged and partially thawed. Until 7.3 is released, sweeping commits still need explicit approval from portmgr to assure that tags can be slipped for potential security issues. For more information what constitutes a sweeping change, see the portmgr web pages.

In preparation for 7.3-RELEASE, the ports tree is now in feature freeze.

Normal upgrade, new ports, and changes that only affect other branches are allowed without prior approval but with the extra Feature safe: yes tag in the commit message. Any commit that is sweeping, i.e. touches a large number of ports, infrastructural changes, commits to ports with unusually high number of dependent ports, and any other commit that requires the rebuilding of many packages is not allowed without prior explicit approval from portmgr after that date.

When in doubt, please do not hesitate to contact portmgr.

In preparation for 7.3-RELEASE, the ports tree will be in feature freeze after release candidate 1 (RC1 )is released, currently planned for February 8.
If you have any commits with high impact planned, get them in the tree before then and if they require an experimental build, have a request for one in portmgr hands within the next few days.

Note that this again will be a feature freeze and not a full freeze. Normal upgrade, new ports, and changes that only affect other branches will be allowed without prior approval but with the extra Feature safe: yes tag in the commit message. Any commit that is sweeping, i.e. touches a large number of ports, infrastructural changes, commits to ports with unusually high number of dependencies, and any other commit that requires the rebuilding of many packages will not be allowed without prior explicit approval from portmgr after that date.

The ports tree is now tagged and partially thawed.  Until 8.0 is released, sweeping commits still need explicit approval from portmgr to assure that tags can be slipped for potential security issues.  For more information what constitutes a sweeping change, see the portmgr web pages.