Made by Mikal – Page 11

Deciding when to filter out large scale refactorings from code analysis

Post author:mikal
Post published:January 3, 2021
Post category:OpenStack

I want to be able to see the level of change between OpenStack releases. However, there are a relatively small number of changes with simply huge amounts of delta in them -- they're generally large refactors or the delete which happens when part of a repository is spun out into its own project. I therefore wanted to explore what was a reasonable size for a change in OpenStack so that I could decide what maximum size to filter away as likely to be a refactor. After playing with a couple of approaches, including just randomly picking a number, it seems the logical way to decide is to simply plot a histogram of the various sizes, and then pick a reasonable place on the curve as the cutoff. Due to the large range of values (from zero lines of change to over a million!), I ended up deciding a logarithmic axis was the way to go. For the projects listed in the OpenStack compute starter kit reference set, that produces the following histogram:I feel that filtering out commits over 10,000 lines of delta feels justified based on that graph. For reference, the raw histogram buckets are:

A quick summary of OpenStack release tags

Post author:mikal
Post published:January 3, 2021
Post category:OpenStack Python

I wanted a quick summary of OpenStack git release tags for a talk I am working on, and it turned out to be way more complicated than I expected. I ended up having to compile a table, and then turn that into a code snippet. In case its useful to anyone else, here it is: Or in python form for those so inclined: RELEASE_TAGS = { 'austin': {'all': '2010.1'}, 'bexar': {'all': '2011.1'}, 'cactus': {'all': '2011.2'}, 'diablo': {'all': '2011.3'}, 'essex': {'all': '2012.1.3'}, 'folsom': {'all': '2012.2.4'}, 'grizzly': {'all': '2013.1.5'}, 'havana': {'all': '2013.2.4'}, 'icehouse': {'all': '2014.1.5'}, 'juno': {'all': '2014.2.4'}, 'kilo': {'all': '2015.1.4'}, 'liberty': { 'glance': '11.0.2', 'keystone': '8.1.2', 'neutron': '7.2.0', 'nova': '12.0.6' }, 'mitaka': { 'glance': '12.0.0', 'keystone': '9.3.0', 'neutron': '8.4.0', 'nova': '13.1.4' }, 'newton': { 'glance': '13.0.0', 'keystone': '10.0.3', 'neutron': '9.4.1', 'nova': '14.1.0' }, 'ocata': { 'glance': '14.0.1', 'keystone': '11.0.4', 'neutron': '10.0.7', 'nova': '15.1.5' }, 'pike': { 'glance': '15.0.2', 'keystone': '12.0.3', 'neutron': '11.0.8', 'nova': '16.1.8' }, 'queens': { 'glance': '16.0.1', 'keystone': '13.0.4', 'neutron': '12.1.1', 'nova': '17.0.13' }, 'rocky': { 'glance': '17.0.1', 'keystone': '14.2.0', 'neutron': '13.0.7', 'nova': '18.3.0' }, 'stein': { 'glance': '18.0.1', 'keystone': '15.0.1', 'neutron': '14.4.2', 'nova': '19.3.2' }, 'train': { 'glance': '19.0.4', 'keystone': '16.0.1', 'neutron': '15.3.0', 'nova': '20.4.1' }, 'ussuri': {…

Rejected talk proposal: Shaken Fist, thought experiments in simpler IaaS clouds

Post author:mikal
Post published:January 1, 2021
Post category:Conference OpenStack Shaken Fist

This proposal was submitted for FOSDEM 2021. Given that acceptances were meant to be sent out on 25 December and its basically a week later I think we can assume that its been rejected. I've recently been writing up my rejected proposals, partially because I've put in the effort to write them and they might be useful elsewhere, but also because I think its important to demonstrate that its not unusual for experienced speakers to be rejected from these events. OpenStack today is a complicated beast -- not only does it try to perform well for large clusters, but it also embraces a diverse set of possible implementations from hypervisors, storage, networking, and more. This was a deliberate tactical choice made by the OpenStack community years ago, forming a so called "Big Tent" for vendors to collaborate in to build Open Source cloud options. It made a lot of sense at the time to be honest. However, OpenStack today finds itself constrained by the large number of permutations it must support, ten years of software and backwards compatability legacy, and a decreasing investment from those same vendors that OpenStack courted so actively. Shaken Fist makes a series of simplifying assumptions…

pngtools, code that can nearly drink in the US

Post author:mikal
Post published:November 22, 2020
Post category:Pngtools

I was recently contacted about availability problems with the code for pngtools. Frankly, I'm mildly surprised anyone still uses this code, but I am happy for them to do so. I have resurrected the code, placed it on github, and included the note below on all relevant posts: A historical note from November 2020: this code is quite old, but still actively used. I have therefore converted the old subversion repository to git and it is hosted at https://github.com/mikalstill/pngtools. I will monitor there for issues and patches and try my best to remember what I was thinking 20 years ago...

Shaken Fist 0.2.0

Post author:mikal
Post published:July 19, 2020
Post category:Shaken Fist

The other day we released Shaken Fist version 0.2, and I never got around to announcing it here. In fact, we've done a minor release since then and have another minor release in the wings ready to go out in the next day or so. So what's changed in Shaken Fist between version 0.1 and 0.2? Well, actually kind of a lot... We moved from MySQL to etcd for storage of persistant state. This was partially done because we wanted distributed locking, but it was also because MySQL was a pain to work with. We rearranged our repositories -- the main repository is now in its own github organisation, and the golang REST client, terrform provider, and deployment tooling have moved into their own repositories in that organisation. There is also a prototype javascript client now as well. Some work has gone into making the API service more production grade, although there is still some work to be done there probably in the 0.3 release -- specifically there is a timeout if a response takes more than 300 seconds, which can be the case in launch large VMs where the disk images are not in cache. There were also some…

The KSM and I

Post author:mikal
Post published:July 19, 2020
Post category:Linux

I spent much of yesterday playing with KSM (Kernel Shared Memory, or Kernel Samepage Merging depending on which universe you come from). Unix kernels store memory in "pages" which are moved in and out of memory as a single block. On most Linux architectures pages are 4,096 bytes long. KSM is a Linux Kernel feature which scans memory looking for identical pages, and then de-duplicating them. So instead of having two pages, we just have one and have two processes point at that same page. This has obvious advantages if you're storing lots of repeating data. Why would you be doing such a thing? Well the traditional answer is virtual machines. Take my employer's systems for example. We manage virtual learning environments for students, where every student gets a set of virtual machines to do their learning thing on. So, if we have 50 students in a class, we have 50 sets of the same virtual machine. That's a lot of duplicated memory. The promise of KSM is that instead of storing the same thing 50 times, we can store it once and therefore fit more virtual machines onto a single physical machine. For my experiments I used libvirt /…

Introducing Shaken Fist

Post author:mikal
Post published:May 28, 2020
Post category:OpenStack Shaken Fist

The first public commit to what would become OpenStack Nova was made ten years ago today -- at Thu May 27 23:05:26 2010 PDT to be exact. So first off, happy tenth birthday to Nova! A lot has happened in that time -- OpenStack has gone from being two separate Open Source projects to a whole ecosystem, developers have come and gone (and passed away), and OpenStack has weathered the cloud wars of the last decade. OpenStack survived its early growth phase by deliberately offering a "big tent" to the community and associated vendors, with an expansive definition of what should be included. This has resulted in most developers being associated with a corporate sponser, and hence the decrease in the number of developers today as corporate interest wanes -- OpenStack has never been great at attracting or retaining hobbist contributors. My personal involvement with OpenStack started in November 2011, so while I missed the very early days I was around for a lot and made many of the mistakes that I now see in OpenStack. What do I see as mistakes in OpenStack in hindsight? Well, embracing vendors who later lose interest has been painful, and has increased the…

A totally cheating sour dough starter

Post author:mikal
Post published:May 23, 2020
Post category:Cooking

This is the third in a series of posts documenting my adventures in making bread during the COVID-19 shutdown. I'd like to imagine I was running science experiments in making bread on my kids, but really all I was trying to do was eat some toast. I'm not sure what it was like in other parts of the world, but during the COVID-19 pandemic Australia suffered a bunch of shortages -- toilet paper, flour, and yeast were among those things stores simply didn't have any stock of. Luckily we'd only just done a costco shop so were ok for toilet paper and flour, but we were definitely getting low on yeast. The obvious answer is a sour dough starter, but I'd never done that thing before. In the end my answer was to cheat and use this recipe. However, I found the instructions unclear, so here's what I ended up doing: Starting off 2 cups of warm water 2 teaspoons of dry yeast 2 cups of bakers flour Mix these three items together in a plastic container with enough space for the mix to double in size. Place in a warm place (on the bench on top of the dish…

A super simple non-breadmaker loaf

Post author:mikal
Post published:May 17, 2020
Post category:Cooking

This is the second in a series of posts documenting my adventures in making bread during the COVID-19 shutdown. Yes I know all the cool kids made bread for themselves during the shutdown, but I did it too! So here we were, in the middle of a pandemic which closed bakeries and cancelled almost all of my non-work activities. I found this animated GIF on Reddit for a super simple no-kneed bread and decided to give it a go. It turns out that a few things are true: animated GIFs are a super terrible way store recipes that animated GIF was a export of this YouTube video which originally accompanied this blog post and that I only learned these things while to trying and work out who to credit for this recipe The basic recipe is really easy -- chuck the following into a big bowl, stir, and then cover with a plate. Leave resting a warm place for a long time (three or four hours), then turn out onto a floured bench. Fold into a ball with flour, and then bake. You can see a more detailed version in the YouTube video above. 3 cups of bakers flour (not…

A breadmaker loaf my kids will actually eat

Post author:mikal
Post published:May 10, 2020
Post category:Cooking

My dad asked me to document some of my baking experiments from the recent natural disasters, which I wanted to do anyway so that I could remember the recipes. Its taken me a while to get around to though, because animated GIFs on reddit are a terrible medium for recipe storage, and because I've been distracted with other shiney objects. That said, let's start with the basics -- a breadmaker bread that my kids will actually eat. This recipe took a bunch of iterations to get right over the last year or so, but I'll spare you the long boring details. However, I suspect part of the problem is that the receipe varies by bread maker. Oh, and the salt is really important -- don't skip the salt! Wet ingredients (add first) 1.5 cups of warm water (we have an instantaneous gas hot water system, so I pick 42 degrees) 0.25 cups of oil (I use bran oil) Dry ingredients (add second) I just kind of chuck these in, although I tend to put the non-flour ingredients in a corner together for reasons that I can't explain. 3.5 cups of bakers flour (must be bakers flour, not plain flour) 2…