Playing with the python prometheus query API

The last few days have been a bit icky around here, with my house apparently proudly residing in the major city with the dirtiest air in the world. So, I needed a distraction... It has also been quite hot, so I wondered how my energy usage was going. I have prometheus monitoring of my power draw, so now seemed as good a time as any to learn how to do some historical querying over the API. I ended up with a python script which can output things like this: "Yesterday had a maximum temperature of 38 and we used 28.36 kwh. The average for similar days is 25.56 kwh." The code is on github if it is of interest to others. I am sure I could push more of this processing down into the prometheus engine, but I couldn't see how to do it today. Hints welcome!

Continue ReadingPlaying with the python prometheus query API

Coming to grips with Kubernetes in 2020: online training

  • Post author:
  • Post category:Kubernetes

There are a few online training resources I've had a play with while learning Kubernetes, so I figure that's worth a quick write up. This is a follow on from my post about Kubernetes podcasts I've tried. I've tried three training providers so far: The Linux Foundation Kubernetes course (LFS258 Kubernetes Fundamentals) is probably the "go to" resource for many people, and is often sold bundled with the certification exams. Unfortunately, it is really terrible. It is by far the worst course I've seen so far. On the other hand, the Linux Academy Kubernetes course is really good. It is flaw is that you have to sign up to Linux Academy, which provides you with all you can eat courses for a rather steep annual fee. Finally, I discovered Mumshad Mannambeth's Udemy courses, and frankly they're excellent. He's put a huge amount of effort into them and it really shows. Even better, with Udemy's regular sales you can pick up his three Kubernetes courses (intro, admin certification, and developer certification) for under $50 AUD. There are even plenty of online quizzes. If I was going to pick a course to try, I'd definitely go with Mumshad.

Continue ReadingComing to grips with Kubernetes in 2020: online training

Hacking on Arlec Christmas lights with tasmota

I'm loving the wide array of electrically certified home automation devices we're seeing now. Light bulbs, sensors, power boards, and even Christmas lights. Specifically Arlec is shipping these app controllable Christmas lights this year, which looked very much like they should work with Tasmota. (Sorry for the terrible product picture, I can't find this product online any more, I suspect Bunnings has sold out for the year?) Specifically, it turns out that these Arlec lights are an ESP8266 which can be flashed with tuya-convert v2 to run tasmota. Once flashed, you can control all of the functions available on the device itself, although there are parts of the protocol I haven't fully understood yet. Let's start off by flashing the device: First off boot your raspberry pi with tuya-convert. I used v2, and I suspect that's important here so make sure you upgrade if you're using something old. Next, put the bud lights into programming mode by holding the button on the control box down until the light strand turns off. Release and the strand should start blinking every couple of seconds. Now run the tuya-convert flashing script. Now go to the tasmota-XXXX essid and enter your wifi details into…

Continue ReadingHacking on Arlec Christmas lights with tasmota

Further thoughts on Azure instance start times

  • Post author:
  • Post category:Azure

My post from the other day about slow instance starts on Azure caused some commentary (mainly on reddit) that prompted me to think more about all this. In the end, there were a few more experiments I wanted to run to see if I could squeeze more performance out of Azure. First off, looking at the logs from my initial testing it looks like resource groups are slow. The original terraform creates a resource group as part of the test and then cleans it up at the end. What if instead we had a single permanent resource group and created instances within that? Here is a series of instance starts and deletes using the terraform from the last post: You'll notice that there's no delete value for the last instance. That's because terraform crashed and never deleted the instance. You can also see that instance starts are somewhat consistent, except for being slower in the second half of the test than the first, and occasionally spiking out to very very slow. Oh, and deletes are almost always really slow. What happens if we use a permanent resource group and network? This means that all the "instance start terraform" is doing…

Continue ReadingFurther thoughts on Azure instance start times

Why is Azure so slow to start instances?

  • Post author:
  • Post category:Azure

I've been playing with terraform recently, and decided to see how different the terraform for launching a simple Ubuntu instance in various clouds is. There are two big questions there for me -- how big is the variation between OpenStack derived clouds; and how painful is it to move between the proprietary clouds? Part of this is because terraform doesn't present a standardised layer of cloud functionality, it has a provider per cloud. (Although, I suspect there's nothing stopping someone from writing a libcloud provider or something like that. It is an interesting idea which requires some additional thought.) My terraform implementations for each cloud are on github if you're interested. I don't want to spend a lot of analysis on the actual terraform, because I think the really interesting thing I found isn't where I expected it to be (there's a hint in the title for this post). That said, the OpenStack clouds vary mostly by capabilities. vexxhost for example seems to only offer flavors that require boot-from-volume. The proprietary clouds are complete re-writes, but are generally relatively simple and well documented. However, that interesting accidental thing -- as best as I can tell, Microsoft Azure is really really slow to…

Continue ReadingWhy is Azure so slow to start instances?

Coming to grips with Kubernetes in 2020: podcasts

  • Post author:
  • Post category:Kubernetes

It has become clear to me that it is time to care about Kubernetes more. I'm sure many people have cared for ages, but the things I want to build at the moment are starting to be more container based now that I am thinking more at the application layer than the cloud infrastructure layer. So how to do that? I thought I'd write down some notes on what has worked (or not) for me, in the hope it will help others. In this post, podcasts. I thought podcasts would be an interesting way to get started with some nice overviews. This is especially true because I'm already a pretty heavy podcast user, so it was easy to slot into my existing routine. Unfortunately this hasn't really worked out. I started with the podctl podcast, but they only ever talk about Red Hat stuff. It is very rare for a guest to not be a Red Hat employee for example. The presenters of this podcast seem to also really dislike OpenStack for reasons they never explain, which is annoying. Then I figured maybe the Google Kubernetes podcast would be better, but it often lacks the depth I am interested in. I…

Continue ReadingComing to grips with Kubernetes in 2020: podcasts

If I Understood You, Would I Have This Look on My Face?

  • Post author:
  • Post category:Book

This book discusses science and technical communication from the perspective of someone who comes from professional theatre and acting. Alan explains how his accidental discovery of the application of theatre sports to communication created an opportunity to teach technical communicators how to be more effective. Essentially, the argument is that empathy is essential to communication -- you need to be able to understand where your audience is starting and and where they're likely to get stuck before you can take them on the journey. Unsurprisingly given the topic of the book, this is a well written and engaging read. The book is nicely structured and uses regular anecdotes (some of them humorous) to get its message across. A detailed and fun read.

Continue ReadingIf I Understood You, Would I Have This Look on My Face?

Prometheus 2.12, query logging, and startup failures on macos

  • Post author:
  • Post category:Prometheus

Prometheus v2.12 added active query logging. The basic idea is that there is a mmaped JSON file that contains all of the queries currently running. If prometheus was to crash, that file would therefore be a list of the queries running at the time of the crash. Overall, not a bad idea. Some friends had recently added prometheus to their development environments. This is wired up to grafana dashboards for their microservices, and prometheus is configured to store 14 days worth of time series data via a persistent volume from the developer desktops. We did this because it is valuable for the developers to be able to see the history of metrics before and after their changes. Now we have a developer using macos as their primary development platform, and since prometheus 2.12 it hasn't worked. Specifically this developer is using parallels to provide the docker virtual machine on his mac. You can summarise the startup for prometheus in the dev environment like this: $ docker run ...stuff... ...snip... level=error ts=2019-09-15T02:20:23.520Z caller=query_logger.go:94 component=activeQueryTracker msg="Failed to mmap" file=/prometheus-data/data/queries.active Attemptedsize=20001 err="invalid argument" panic: Unable to create mmap-ed active query log goroutine 1 [running]: github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff9917af38, 0x15, 0x14, 0x2a6b7c0, 0xc00003c7e0, 0x2a6b7c0) /app/promql/query_logger.go:112 +0x4d2 main.main()…

Continue ReadingPrometheus 2.12, query logging, and startup failures on macos

The wonderful world of machine learning automated lego sorting

Inspired by Alastair D'Silva's cunning plans for world domination, I've been googling around for automated lego sorting systems recently. This seems like a nice tractable machine learning problem with some robotics thrown in for fun. Some cool projects if you're that way inclined: Sorting 2 Metric Tons of Lego A lego sorter using tensorflow This sounds like a great way to misspend some evenings to me...

Continue ReadingThe wonderful world of machine learning automated lego sorting

End of content

No more pages to load