Flock to Fedora 2016

Keynote: State of Fedora

Flock Day 1 started off with a bang, our very own Fedora Project Leader, Matt Miller took the stage for the morning keynote and discussed the current state of Fedora, where we are, where we're going, ongoing work and current notable Changes with work under way.

One of my favorite take aways from this talk was one about contributor statistics that are gathered based on contributor activity as it is represented within the Fedora Infrastructure via fedmsg and datagrepper (datanommer). The statistics showed that there are over 2000 contributors, of which roughly 300 do 90% of the work (which sounds odd, but statistically this is actually better than average) and of the group that does 90% of the work only about 35% of them work for Red Hat. I'm a big fan of these kind of numbers because it reinforces that Fedora is in fact a community driven project that Red Hat is simply a participant and sponsor of.

Flock 2016 Keynote State of Fedora slides

Introducing Fedora Docker Layered Image Builds

Next time slot that I attended was my presentation on the Fedora Docker Layered Image Build System here I introduced something I've been working on for quite some time with various upstream projects of technologies that come together to form this system. Before diving into the new service I went on a brief history lesson about what containers are, what they are in the context of Linux, and various implementations of which Docker is simply one. The main reason I like to start there is to level set that we have hopes to support all kinds of Linux container runtimes and image builds but we must start somewhere and with Docker being the most popular it makes sense to target it first. (You'd be surprised how often the question of supporting other image formats comes up)

In an attempt to make sure there were no gaps in knowledge of everyone in the room for my presentation I did a quick overview of what specifically Docker is, how containers are instances of images, and how images themselves are most commonly built (Dockerfile). We then progress into concepts of Release Engineering and why this is desirable, as outlined in an article I wrote for OpenSource.com recently. From there we traversed into the wild world of distributed container runtimes and orchestrators, most notably OpenShift as that's a core component of the Layered Image Build Service. We also discussed components used within the Docker Layered Image Build Service such as atomic-reactor, osbs-client, and koji-containerbuild. The last of which enables for the workflow using fedpkg for layered image builds for Fedora contributors just as they are used to for RPM.

I then did a demo, that of course failed (as per the Demo Gods) but was able to show a previously successful build.

Introducing Fedora Docker Layered Image Build slides

Getting New things into Fedora

In recent past there has been a general communications break down between developers and Release Engineering, this has resulted in some issues integrating net-new deliverables within the Fedora Project. This presentation discussed the process by which new changes should come in, the timelines that things should be accepted by, and the various Release Engineer Tools that need integrating with.

However, there was admission that the documentation could be better about these items and the Release Engineering tools could be more approachable for outsiders in order to help with the process of on-boarding new changes into the processes and tooling. These items have shown improvement in the past year with further improvements planned.

There was a lively discussion of ways to make this better and I look forward to seeing positive movement come as a result.

Hacking Koji for fun and Profit

In this session, tip and tricks for hacking on the Koji build system were the focal point. Discussion about what Koji is, who uses it, and why someone might want to hack it was explored. Then an overview of the major components of Koji were presented in an attempt to give potential developers an idea of where to look in the code depending on what component they were trying to augment or supplement. From there a quick example of the Python API was covered as an example of how to get started, including reference for a more advanced example contained within the koji code itself was offered. Next up was a advanced CLI walk through that showed how to call directly to the XMLRPC API just as you can via the Python API.

There was a section of the session focused on the Koji Hub which is the user facing component including how to theme the web UI, change user policy, and how to write plugins that can add functionality to Koji via new API calls, policies, and callback hooks.

Next up was discussion of Koji Builder plugins that can add the ability for Koji to produce new types of Build Artifacts.

Finally, how to actually clone the git repository and then build locally a version of the modified code was covered.

During the Q&A portion there was a discussion of how difficult Koji can be to deploy and that it would be nice if there was a way to get up and running quickly for hacking purposes. Something that was completely automated and not necessarily production ready would be desired. There was also lively discussion about the future of Koji and the iterative improvements already made in refactoring the code as well as plans for more. Originally there was a grand plan for a "Koji 2.0" that would be a complete rewrite but as time has gone on that has proven too lofty of a goal to realistically achieve so the more iterative approach is being taken.

Hacking Koji for Fun and Profit slides

Containers in Production

Dan Walsh discussed running Containers in Production, a topic that is hot on many people's mind as container technology races into the mainstream as fast as OpenStack did before it. This session discussed various means of container runtime execution, including that of Docker and it's daemon. This included various aspects the Docker daemon's strengths and weaknesses and why alternative execution methods might be desirable or at least worth considering for Production workloads and environments. Various aspects such as storage configuration, non-privileged runtimes (security), remote inspection, fault tolerance, and systemd integration were discussed.

In this presentation was a strategy for running production ready containers using runC for execution of Open Container Initiative (OCI) compliant container images (such as Docker Images).

https://github.com/containers

https://github.com/projectatomic/skopeo

Fedora's MirrorManager: now and in the future

The session about MirrorManager was extremely informative, covering various aspects of the project, a brief overview of the history then diving into current features, limitations, things we're trying to do in the future to improve and enable the mirroring of new artifacts.

There were plenty of items that I would like to follow up on as there's so much about content mirroring that I don't currently understand.

I sadly did not take nearly as good of notes during this session as I had hoped to. I highly recommend anyone interested in the topic of content mirroring to watch the recorded session for more information.

Fedora ARM State of the Union

Peter Robinson gave a presentation about the current state of Fedora ARM including both armv7hl and AArch64. At the start of things he requested that questions about specific dev boards be held to the end because he would have a section in the session dedicated to that. Exploration of the trials and tribulations of bringing new hardware to life was interesting (at least to me) as there's so many things that we in the pre-existing hardware platform world take for granted. There's many things about the ARM world and boot firmware that make things difficult because of lack of standardization around the developer board boot methods beyond just the standard trouble of bringing up new hardware that doesn't yet have support for everything necessary in the kernel. Beyond the kernel is the compiler toolchains and programming language tooling that needs added support for new architectures such as ARM, various points of this were discussed with examples of areas where people in the Fedora community have stepped up to help (Haskell SIG being one great example).

From there discussions of various developer boards spiraled off into the weeds of things that I definitely don't understand about the finer points of ARM board "bring up" but it was interesting to listen to the state of things and take notes of things to learn about.

University Outreach - New Task or New Mindset?

Justin Flory and Jona Azizaj presented about the history of the University Involvement Initiative, the struggles met with attempting to expand it's adoption and further reach, and eventually it's decline. This session was a call to arms for community members with ties to Universities either as active students or Alumni to help bring this initiative back to life. The main idea behind all of this is that we would like to help foster the open source community by bringing an active student population into it's ranks. There was a lot of positive feedback and interest shown during the session and I have high hopes for the future of the initiative.

Fedora Engineering Team Dinner

While not on the Flock schedule, this was a personal highlight for me as a member of the Fedora Engineering Team because we are a geographically dispersed team that lives and works from all corners of the planet. As such, we rarely get the opportunity to all be in the same place, at the same time, and in a social setting (as opposed to getting work done). It was great to be able to sit and chat with colleagues and discuss both work and non-work topics and get to know them better on a more personal level.

The main take away: I love my job, I love my team, and I love my company.

Kirk, McCoy, and Spock build the future of Fedora

Matt Miller took us on a Star Trek themed adventure that lead to the use of the Kellog Logic Model applied to Fedora Initiatives and how each Working Group (WG) or Special Interest Group (SIG) could use this model as a means to help drive their goals as well as frame their over all initiatives to others, including the Fedora Council and FESCo. The session slides were covered rather quickly and then discussions and questions broke out about how we could use this for various things and/or just general questions about the logic model.

The Fedora Modularity Logic Model was an example where this is already being used within the Fedora Project with success.

Modularity: Why, where we are and how to get involved

Fedora Modularity is a new initiative that is focused on re-thinking how we think of the way Linux distributions are composed. Instead of as a pile of software packages, it could be a grouping of modules that can be managed as disjoint units and lifecycle managed independently of one another.

Background on the topic leads back to the Rings Proposal (a part of Fedora.next), where we think about the distro as a set of rings and the center of rings the central point of the operating system is the most curated components of the operating system and as you get further from the center you can have less and less curation.

However, as time went on you have less and less correlation such that the Rings analogy doesn't really fit. Example, any given package can change over time or need a different version in a different use case or scenario.

Different use cases, a new website with the latest technologies vs an ERP system where you want different lifecycles or different "aged" or different levels of "proven" technologies. This is the problem that modules hope to solve.

What is a module?

• A thing that is managed as a logical unit

• A thing that promises an external, unchanging, API

• A thing that may have many, unexposed, binary artifacts to support the external API

• A module may "contain" other modules and is referred to as a "module stack"

Base Runtime (Module Stack)

• Kernel (module)

• userspace (the interface to userspace, coreutils, systemd, etc)

  • There built requirements are not part of the module, but simply a build requirement.

modulemd: Describe a module

• yaml definitions of modules, standard document definitions with "install profiles"

• install profiles

• definition of components included in a module

There was plenty of discussion around these topics and suggestion that people attend the workshop the following day.

Factory 2.0

As with all things in technology, we want to constantly move faster and faster and the current methods by which we produce the operating system just won't scale into the future. Factory 2.0 is an initiative to fix that.

The presentation kicked off with a witty note that we have entered the "The Second Eternal September," GitHub and how node.js has changed how people fundamentally expect to consume code.

Dependency freezing has become common practice these days because of node.js and rubygems communities impact on developers.

pip freeze > requirements.txt

• ruby bundler

• nixOS

• coreOS

• docker and friends

Brief overview of Fedora Modularity was given for those who didn't make it to Langdon's session on the topic.

Matt Miller started with Fedora.Next -> Rings, then Envs and Stacks, Red Hat now funding a team to accomplish this.

Backing up first to discuss how not to throw things over the wall. In past there's been discussions about how to articulate "Red Hat things" in the Fedora Space. Ralph Bean (our presenter) works for a group in Red Hat called RHT DevOps.

There are analogous groups within Red Hat and the Fedora Community:

Fedora Packagers -> RH Platform Engineering

Fedora Infra -> RH PnT DevOps

What Factory 2.0 is not: a single web app, a rewrite of our entire pipeline, a silver bullet, a silver platter, just modularity, going to be easy.

"the six problem statements"

• Repetitive human intervention makes the pipeline slow

• unnecessary serialization

• rigid cadence

• artifact assumption

• modularity

• dep chain

"If we had problems before, they're about to get a lot worse" (Imagine modularity without Factory 2.0)

Would like to use pdc-updater to populate metadata tables with information about dep chains, we would then use that information with other tools like pungi but also with new tooling we haven't even thought of just yet.

Unnecessary serialization makes the pipeline slow, one big piece we will need to solve this is the OpenShift Build Service (OSBS). We're going to need to use an autosigner.py to get around new problems (assuming we "go big" with containers).

Automating throughput, repetitive human intervention makes things slow. Builds and composes. An orchestrator for the builds and the composes, best case scenario is that things are built and composed before we ask for them.

Atomic Host Two Week is kind of a case study that we should learn lessons from in order to merge the changes needed back into the standard pipeline instead of the parallel pipeline that was spawned.

Flexible Cadence, The pipeline imposes a rigid and inflexible cadence on "products". Releases related to the previous point about Automating Releases, "the pipeline is as fast as the pipeline is".

EOL: think about the different EOL discussions for the different Editions. Beyond that - a major goal of modularity is "independent lifecycles"

"I want to be able to build anything, in any format, without changing anything" (not possible) but we can make the pipeline pluggable that will make it easier over time to add new artifact types to the pipeline.

"The pernicious hobgoblin of technical debt" as Ralph called it.

Ways we can do better and refactor:

• Microservices (consolidate around responsibility)

• Reactive services

• Idempotent services

• Infrastructure automation (Ansible all the things)

Docker in Production

The Docker in Production session was a very brief walk through of how you can go from your laptop to a production environment. This effectively boiled down to best practice for how to "containerize" your application properly, ways to build docker images and tagging schemes that you can (or should) use, a distribution mechanism for the images, and finally a distributed orchestration framework such as Kubernetes, OpenShift, or Docker Swarm.

Pagure: Past, Present, and Future

Pagure is a git forge.

Old version was very simple: there were three repos per project: source, tickets, and pull requests. Recently got a new UI (thanks to Ryan Lerch).

Forks, pull requests. (A very GitHub style workflow).

If you want to run your own pagure, all you need is the web services and the database. If you'd like all the bells and whistles, you'll then need to add mail server (pagure milter), pagure eventsource server, gitolite, and a message bus.

Doc hosting (fourth git repository for a project, optional), in the future considering doing something similar to GitHub Pages.

"Watch" repo, to get notifications for a project you're not directly involved in or to opt out of notifications for a project you are directly involved in.

Roadmap in the Issues tab in the UI for milestones and arbitrary tag filtering.

Issue templates, delivered by markdown files in the issues git repo. Also, can set a default message to be displayed when someone files a new pull request.

Diversity - Women in Open Source

The session on Fedora Diversity began with a lot of wonderful information about the initiative and I have outlined to the best of my ability focal points of those slides here.

• Started roughly a year ago

• No exists an official Fedora Diversity Adviser

• Myths

  • Women are not interested in technology

  • Women can't to programming

  • Men developers are mote talented than women

  • There is no work-life balance for women who work in the tech industry

  • So on and so on ...

• Facts

  • Women in Technology (Mothers of Tech - BizTech)

    • Ada Lovelace (Creator of Programming/Computational Machine)

    • Heddy Lamar (Frequency Hopping)

    • Admiral Hopper (Created COBOL)

    • Many more ...

  • Women are very creative, versatile, powerful, and intelligent

  • Diversity increases success

• Initiatives

  • Grace Hopper Celebration of Women in Computing

  • Women in Open source Award

  • Outreachy

  • Google Summer of Code

  • and many more

• Gaps

  • Lack of knowledge, encouragements, support, and time commitment

After the slides were done, the session turned into effectively a giant round table of people telling stories of how they've been successful because of diverse teams, reasons they think that women and other groups of people are currently under represented in Fedora and Open Source, ways they feel we can increase diversity, and methods that could be used to target various under represented groups in the Global Open Source community.

The GNOME Outreachy program was also discussed as a great example of a program working to move things in the right direction around the topic of how we can try to actively improve our community and the open source community at large.

I hope to be able to participate in some of the take aways from these discussions as they are put into action.

Testing Containers using Tunir

tunir is a simple tool that will spawn a virtual machine or several virtual machine and then execute arbitrary commands and report success or failure of the commands based on the exit code of the command. You can also make commands blocking or non blocking and tunir has support for Docker images as well as support for spinning up a kubernetes multi-node cluster in order to test containers "at scale". The presentation was short and to the point with plenty of demos showing how easy it is to get started using tunir. Also, tunir is the testing component behind Fedora AutoCloud.

Cruise Krakow

In the evening of Day 2 the Flock participants had the unique opportunity to dine on the Vistula River and take a small tour up and down the river for some site seeing. It was a beautiful scenic way to wind down with fellow Fedora Flockers after a full day of sessions and technical discussions.

Lightning Talks

Day 3 kicked off with Lightning Talks, I presented one myself about a small project I've been working on titled Loopabull which is an event loop driven Ansible playbook execution engine. There were also plenty of other wonderful lightning talks covering topics such as Fedora Marketing, OpenShift provisioning on Fedora with Amazon Web Services, Fedora CommOps, dgplug, and so much more.

Automation Workshop

The automation workshop was kind of an anti-presentation session as the session leader wanted this to either become more of a hacking session of a problem solving session. As such, ad-Hoc discussions and work done on automation issues in the various areas of the Fedora Infrastructure occurred and people broke off into smaller groups within the room to work or solve problems.

OpenShift on Fedora

This session was about running OpenShift on Fedora using the latest and greatest features of OpenShift, most notably the new component called oc cluster up which is an auto-deployment provisioning tool built directly into OpenShift as of version v1.3+ which allows for the automatic creation of a clustered environment. The entire session was provided as a very well documented walk through and the link is below.

OpenShift on Fedora Guided WalkThrough

Building Modules Workshop

The Modules building workshop came together as a hybrid approach of some presentation, some discussion, some demo, and some "follow along" workshop style. This was a lot of fun and incredibly informative, there was lively discussion about aspects of a module definition (for me it was mostly about trying to wrap my head around everything, and the session hosts were very accommodating).

There were many notes taken during the session and they were preserved in an etherpad instance but in the event that it gets lost in the ether over time I have exported it's contents to my FedoraPeople space and it can be viewed here.

Brewery Lubicz

Next up is the evening event which was hosted in a brewery complete with wonderful catering.

As per the schedule:

A feast and beer tasting awaits us at Browar Lubicz, a recently restored brewery. The brewery dates from 1840 and has been brewing beer almost continuously, even during nationalization in the 1950s. Restored in September 2015, the brewery is a high point of a trip to Krakow.

What we do for Docker image test automation

I attended this session for about 45 minutes but it quickly became apparent that the other participants were very new to Docker and taskotron in general and the session would therefore be very introductory in nature so I departed to join a workshop elsewhere. This session was by no means bad and I think anyone who is new to the topic of Docker or taskotron and is interested how these two things are being used together in order to test Docker Images should absolutely have attended or should watch the recording on YouTube after the fact.

Server SIG Pow-Wow

A lot of things are changing in the Fedora Project, particularly for modularization. This session was by and large a collaborative brainstorming and planning session for how to take advantage of the new initiative and how to adapt properly. RoleKit became a focal point of discussion as well as Ansible and potentially an integration with the two. Aspects of the discussion related back to the Fedora Formulas proposal that unfortunately didn't get off the ground at the time.

The session leader graciously took notes and has plans to post a write up.

Informal Friday Night Shenanigans

Friday night a group of us Flockers took to the streets of Krakow City Center in order to take in as much of the local cuisine, culture, and sites as we could on our last night in town (at least it was the last night for some of us). This was a really great outing and I had the opportunity to make some new friends within the Fedora Community that I had yet to meet in person. It was a wonderful way to close out an amazing event.

I look forward to Flock 2017!

Until next time...