Scroogie Boy

Wed, 18 Feb 2026 07:55:47 -0500

If AI-written business applications are supposedly taking over from SaaS, who is operating them? Who is making sure they run, are backed up, can be restored in the case of disaster… Or does business continuity not matter anymore?

Ikea Boaxel impressions as a Elfa shelving user

Fri, 26 Sep 2025 06:55:02 -0500

As a long-time Elfa shelving user, I was looking for a cheaper option in less demanding scenarios. After a bit of searching online, I found the Ikea Boaxel system. Here are some of my notes after building a wall storage solution using the Boaxel system:

First of all, the two systems differ dramatically in cost – given that the Ikea Boaxel is about one third the price of equivalent Elfa shelving, we shouldn’t go into this with the expectation of getting the same thing. These systems target very different price points.

One of the ways that the Boaxel system achieves its lower price point is by being flimsier than Elfa. If you intend to put heavy things on the shelves, this is not the system for you. Ikea don’t seem to quote the system load-bearing capacity in their specifications, but it is less sturdy in all ways compared to Elfa. Not that this is inherently a flaw – we don’t all need to store big heavy bins on our shelves – but it is an important deciding factor.

Beyond being generally flimsier, the way that the Boaxel system transfers loads to the wall is quite different: while Elfa shelving transfers loads primarily to the overhead suspension rail, the Boaxel system puts the load on the (many – one about every 10 inches) fasteners on the uprights. The uprights will flex if they aren’t fully fastened to the wall, so the top rail is mostly an alignment tool than a load-bearing part of the system.

The sheer number of fasteners needed in the Boaxel system may also be a turn-off: while Elfa shelving allows you to mount the top rail, then slide the uprights around freely (handy if you like to reconfigure your shelving), Boaxel top rails and uprights must all be fastened to the wall. Reconfiguring the location of the uprights in a Boaxel system is a lot more work. In US stick-frame construction, this means making (and patching on removal) a lot of holes in the drywall for the fasteners.

Other than the MDF “wood” shelves (which I didn’t try), the various shelving options in the two systems don’t have direct equivalents. The wire shelving in the Boaxel system is much lighter-duty than the Elfa ones. The stamped steel shelves have no equivalent in the Elfa system and are a handy addition.

The tabs that mount shelves to their supports seem to be an especially weak part of the Boaxel system. They are small and easy to bend out of shape. This isn’t a fatal flaw, but something to be careful about when installing the shelves – if something doesn’t fit perfectly, brute-forcing it will likely end up bending some tabs and making things worse. You need to be more careful when installing Boaxel shelves while the Elfa wire shelves tend to actually require a little force to clip in properly.

Finally, Boaxel does not have the plastic rail covers to make the system look neat and tidy. I never use them with my Elfa shelves, but if this matters to you, the Elfa shelving system will look a lot more “finished” than Boaxel will. You can cover the suspension rail and hide the fasteners in an Elfa system. The fasteners are all visible in the Boaxel system. The brackets between shelves arguably require less beautification (because they are so slim), but the ends aren’t smooth and are covered with hard-to-remove labels.

Why not develop for Apple platforms?

Mon, 09 Jun 2025 11:57:44 -0500

On the occasion of WWDC starting today, I thought that I’d write some thoughts down as to why, even as a fan of Apple platforms, I still would not want to be a developer for those platforms.

The old days

I have to admit that I am not a long-time Apple fan in the “bleed six colors” sense. Back in the 1980s, I was more of a hardware hacker with too limited a budget to ever seriously look at Apple products. In the 1990s, Apple had drifted into comparative technical obsolescence, so it was really hard to get excited about the mac. I occasionally used one at work, but world had more or less caught up with the basics of the user experience and the classic MacOS technical underpinnings were stale. There were a multitude of operating systems that were technically more exciting and good enough from an UX perspective, so why pay attention to the also-ran in the market?

Getting on board

Come the 2000s and OS X, things have changed. The new Intel macs provide an on ramp to the platform with a built-in escape hatch (if all else fails, we could always fall back to Windows), so I’m one of the early Intel mac adopters (yay for the white polycarbonate MacBook!) and haven’t looked back – as a user – since. Professionally, though, it has never made sense: the kind of software I worked on was always technical, but never important enough to the customer to drive a platform switch. It was always “put your thing on our corporate standard PC”. Later on, web-delivered applications didn’t really care about the client computer – as long as there was a browser, it would run.

During this whole time, the mac remained a great developer machine – whether developing for Windows (in a VM) or Linux servers (the UNIX underpinnings of modern macOS help a lot), the mac was a great place to work, but never the development target.

Looking at the applications I used on the mac, I could see that there was a really awesome combination at play: a solid operating system, frameworks like AppKit that brought fundamental improvements to the developer experience and human interface guidelines that you could create usable, productive software without a massive investment in programming or design effort. I still didn’t have a work reason to make anything for the platform, but the productivity advantage for technical applications seemed real. The barrier to entry for mac application development was pretty low – tempered by the then-small market.

Enter the iPhone

The iPhone changed things fundamentally. Yes, in all the ways we know, but also it changed the expectation of what an application should look like. Over the next few years, the expectation of highly-designed, highly-polished (as opposed to usable) negated the productivity advantage. This permeated other software platforms, of course, but it shifted the baseline of what people expect from software (even line of business enterprise software) in an expensive way. Thankfully, iOS 7 and some of the more recent UI paradigms (although we may see something different at this year’s WWDC) put a damper on the design arms race around some of the more expensive trends, but I get the feeling that the baseline application for an Apple platform got more expensive to develop over the years.

Simultaneously, the iPhone – by virtue of its scale – devalued third-party software. With millions of installations, even complicated software can be sold pretty cheap. However, the expectation that all software is cheap has made low-volume applications extremely hard to sell for what they cost to make, even if they are worth it (i.e., bring commensurate user benefit).

The iPhone also introduced a lot of walled-garden practices. People like to complain about the revenue split requirements, problematic as they are, but they aren’t really the fundamental problem. I think that the real problem is that Apple introduced itself as a gatekeeper between the developers and their customers. They gave themselves the right to deem an application unfit for sale, based on ever-changing interpretation of App Store rules that now means that anybody thinking of entering the Apple ecosystem as a developer needs to factor in that their application could be rejected for… well, no good reason.

In plain terms, it has dramatically increased the risk of developing for Apple platforms. When the platform owner has the ability to unilaterally block applications from sale for nebulous reasons that can’t be determined in advance – I specifically do not mean gross violations of their basic App Store rules, I mean when reviewers become arbiters of what functionality is “good”, sufficient and allowed – with little or no recourse.

Under these circumstances, who wants to spend a non-trivial (and increasing) amount of development effort making something that the reviewer of the day may decide has no place in the walled garden? Compared with the acceptance roulette (repeated with every new submission), the rent-seeking demand for a revenue percentage pales in comparison. As an outsider looking at the Apple platform market, rent seeking is is a fact of life one can plan for (although it does make the market less attractive on the whole). What is not manageable is the uncertainty. Not knowing whether an application will be allowed until you spend significant resources and opportunity cost to build it is going to remain the biggest barrier to entry for new developers that aren’t already committed to the platform.

I’m sure that Apple feels that the intrinsic value of access to that walled-in market is motivation enough, but the truth is that small businesses growing organically have only so much tolerance for risk. Add to that the expectation that all software is now “cheap”… Why would anybody bother?

This is something that Apple has the ability to address. Whether they want to or not is another matter.

P.S.: I realize that mobile device management (MDM) is one way to tackle this problem for custom/proprietary Enterprise applications, but there is a huge chasm between being able to sell to anyone via the Apple (or other, in the EU) App Store and participating in the Enterprise sales market. To some extent, it is just trading one gatekeeper for another. Neither is particularly attractive for small ISVs.

Video creation — the evergreen money pit

Fri, 06 Jun 2025 23:10:55 -0500

Generating decent-quality video content (e.g., for YouTube) is a definite test of one’s priorities… “Decent” these days means 4K footage with some lighting and a good quality mic. The storage and processing requirements aren’t trivial. Now I see why people get excited every year when the new macs come out. For a developer, the speed ramp is a gentle slope. For video professionals, it is a steeper slope (at significant cost!).

I do wonder how many small creators actually archive their projects? It seems like the archival storage needs could get really expensive really fast. Maybe most people don’t care and just discard the intermediate files? To be fair, there isn’t a ton of incentive for normal people to revisit past projects. So, maybe the answer is to be realistic about priorities and treat this stuff more as throwaway artifacts. Just keep the final video in case something bad happens with the hosting provider…

Canon PIXMA Pro-200s black & white printing update

Thu, 29 May 2025 09:52:04 -0500

As a follow-up to my Canon PIXMA Pro-200s first impressions, I have to admit that I was wrong – well, wrong in one very specific way. But, before I talk about that, I do need to cover what hasn’t changed: I still really like this printer and it is fantastic for color prints and color-managed black & white prints.

So, to get to the part where I was wrong, I did some experiments with the Canon “Black & White Photo” mode on more papers and this is definitely a mode that is very paper-sensitive. I wouldn’t exactly say that I’m seeing metameric failure by the dye ink, but I definitely see different tints on different papers – which then are magnified by different lighting. I am sure that there is some degree of ink metamerism in what I’m seeing, but the main effect is more of a paper-specific color tint in black & white mode.

You could call it a “problem” in that it does limit paper choices somewhat and requires a bit more experimentation to get things right, but it’s a problem I am willing to live with in return for the brilliant color prints. It is also a problem that people work around by printing in color mode with a good paper-specific ICC profile. I remember watching YouTube videos by FotoSpeed (a paper supplier in the UK) where they strongly recommended using custom profiles to make black & white prints. At the time, I thought it was a little incongruous, given how others like the black & white print mode. Having done some test prints, I think I understand this a little better: the black & white print mode is a fairly paper-independent one-trick pony when the ICC profiles are custom to each paper specifically. It’s possible that I could get better results by using the advanced adjustments in Canon Professional Print & Layout, but that seems like a lot more work than just using a custom profile.

The experiment:

I printed the same black & white picture on four different papers (three Canon papers and one Red River paper) using “Black & White Photo” mode in the Canon driver.
After drying overnight, I laid them out on the floor alongside an 18% grey card.
I photographed them under different lighting conditions, processed the pictures in DxO PhotoLab to set the white balance according to the card, de-skew the picture and turn the vibrance dial to the maximum.

Each picture has the same papers in the same order, clockwise from top left:

Canon Pro Luster Photo Paper
Canon Matte Photo paper
Red River Aurora Art White 300
Canon Pro Premium Matte Photo Paper

Turning the vibrance dial to the maximum accentuates (in a way that isn’t apparent to the naked eye) some of the color discrepancies in the prints. Repeating the exercise with different light sources also shows some of the different responses:

Natural daylight
[Incandescent light bulb][https://flickr.com/photos/cwirving/54522028850/]
Some random old CFL bulb
[A consumer-grade “daylight” LED bulb][https://flickr.com/photos/cwirving/54521927743/]
A consumer-grade “warm white” LED bulb

I don’t think I can give any meaningful quantitative results from this tiny experiment using comparatively inaccurate instruments, but my main conclusion (other than “make sure you have decent lighting”) is that the black & white prints are sensitive to both paper and the light source used to illuminate them.

It’s also interesting that the Red River Aurora Art paper performs very differently in this print mode – circling back to the FotoSpeed comments earlier, this could be why they prefer printing with a custom profile: the black & white mode “special sauce” depends on the Canon paper used. A subsequent experiment on the same paper using the Red River-provided custom profile does seem to confirm this (the print has a much more neutral tone and density).

Adobe shenanigans of the week

Tue, 20 May 2025 12:41:50 -0500

I ditched Adobe Creative Cloud a few years ago (for a combination of DxO PhotoLab and the Affinity suite) and every so often wonder whether I just traded one treadmill for another… Then there are news stories of these shenanigans by Adobe that remind me how expensive that Adobe treadmill was.

And, of course, I don’t have to upgrade PhotoLab every year… But I do enjoy the additions so far, so I have been doing so.

Canon PIXMA Pro-200s first impressions

Mon, 12 May 2025 14:36:11 -0500

First of all, I’m just a hobbyist — not a professional photographer. I have no intention of selling prints, archiving them for future generations, etc. So, with that said, the entry-level “pro” printer from Canon is surprisingly good.

I think that it’s pretty easy to come away from YouTube and the Internet in general thinking that photo printing is for pigment inks, period — that dye inks, as used in the Pro-200, are so inferior that they aren’t worthy of consideration, especially if you print black and white pictures or print on matte paper. Well, that’s clearly not true…

The truth is far more nuanced and the likes of Keith Cooper do a great job of explaining the finer details, but you could easily believe that something like the Pro-200s (or it’s predecessor, the Pro-200) give up a lot once you stray from color prints on glossy paper. Maybe they do give up something… but that doesn’t mean that the results aren’t already pretty spectacular. Looking at test target prints across a few representative Canon papers was eye opening — the predicted drop-off in quality when printing on matte paper or black and white print metamerism failures didn’t materialize. I’m sure that I’ll find issues as I try other papers, but the results on the handful of Canon and Red River papers I have tried so far are very nice.

Everything is relative — compared with ten- or twelve-ink printers, this is probably a clumsy toy — but I think that an entry-level “pro” printer exceeds hobbyist needs quite handsomely. To be clear, I don’t think that this is specific to the Canon Pro-200s. It likely is true for the whole class of competitors at this level (like the Epson ET-8550).

So, in short, first impressions are very positive.

Desired state and DevOps

Wed, 23 Apr 2025 11:31:57 -0500

TL;DR

This has turned into a long-winded rant, so what point am I trying to make here? Well, a couple of them:

Whenever one says “desired state”, there should be some form of reconciliation, not just unidirectional change propagation.
The deterministic portion of the reconciliation should be fast, relying on reproducibility to deduplicate time-consuming steps. Without the needless slow “build” steps, the performance of the overall reconciliation process becomes easier to focus on and optimize.

The rant

Related to my example yesterday where the desired state of a complete SaaS control plane can be quite successfully be stored in a Git repo, I think it is worth taking a moment to unpack the intent of such an implementation and see how implementations frequently fail to live up to that intent. In short, when saying that a repo is the desired state of X, we imply that something will reconcile the actual and desired states. In Kubernetes terms, a reconciliation loop.

Build systems don’t reconcile things

My experience has been that reality on the ground is usually not so nice – the old-fashioned “build system” roots of DevOps infrastructure leads everyone to treat this as an open-loop state machine rather than a closed-loop control system. This is where naming doesn’t help: “GitOps” as a term falls short of the full intent, quite possibly because it has been captured by vendors (just like “observability” has). Sure, the common usage of the “GitOps” term captures the idea that the desired state of the system is present in a Git repo and that something will make the real world reflect it, what it doesn’t capture is how the real world will be changed. Because of the industry legacy of build systems, there is an assumption that the only source of changes is the repo – everything else flows unidirectionally from there. That’s simplistic, but false, assumption.

The long shared history of “I have new source files, therefore I must build and deploy new binaries” is comfortable, but it isn’t really the job to be done. The job to be done is to ensure that the deployed binaries reflect the source code. It doesn’t matter why they don’t currently match, the situation needs to be remedied (by, absent any optimizations, building and deploying the code). So, we have a lot of tooling that was created for a subset of the job to be done (the hammer), leading people to treat everything as a nail.

People set up build systems to be inefficient

Aside from these build systems' inability to recognize and recover from changes that don’t follow the strict state machine they were coded to implement, the heritage of applying them to flakey technologies leads to rigid and inefficient patterns. If merging a PR to the main branch means that it is code that we want to see in production, why do I see so many teams run all the tests, security scans, etc. again after the merge? They should be prerequisites to the merge (and often are), but teams see the “build” as the one true pipeline that runs as continuous integration the only source of truth, ignoring all checks that happened prior.

The sad thing is that the push towards immutable build artifacts and reproducibility has been underway for many years. In a world where “building” the same code produces the same results forever, there is no need to build it more than once. Those results can be cached for efficiency. The reason I put “build” in quotes is because so many of the time-consuming transformations that happen along the path from checking in code to having an updated running system are now reproducible, deterministic, transformations. These aren’t just code compilation steps – unit testing, bill of materials determination and so forth are, if done properly, immutable. The work that remains based on external state, such as determining which software vulnerabilities are present in the bill of materials today, is limited.

We shouldn’t live in a world where all the same expensive steps are performed over and over. Not just from resource efficiency perspective, but also from an operational perspective: if the reconciliation from code to deployed system is fast, there only needs to be one way to roll back code, for example – just update the main branch to reflect the desired version. So many of the special cases teams are forced to include in their operational processes are just there to work around inefficiencies that have been carried forward from decades-old infrastructure concepts. While I’m not in love with the implementation, I think that Dagger is a good example of somebody rethinking the problem in a modern setting.

Goldilocks SaaS control planes

Tue, 22 Apr 2025 16:55:06 -0500

I find that reading the Building Multi-Tenant Architectures book leads to a lot of nodding in agreement, with the odd raised eyebrow. One moment that struck me relatively early on is that Tod Golding is not letting the reader get carried away with the features of the SaaS system they want to build – the control plane of the system is fundamental to actually having a sustainable product. We’re in violent agreement there. What led to the raised eyebrow is the way in which the reader easily can go down a rabbit hole of overshooting the non-functionals of the control plane by building it with the same technology and mindset as the application plane itself.

Of course, there are cases where the control plane and its various components deals with massive volumes or has challenging latency requirements. Those cases will require a carefully-crafted high-performance control plane, sure… Most likely, there are pockets of demanding functionality in the control plane (e.g., the hot path through identity management, application observability, etc.), but most of the control plane deals with relatively small volumes of data with infrequent changes. Not everyone intents to scale to the heights of large-scale business-to-consumer (B2C) SaaS, so building a control plane implementation that meets those needs out of the box is gross overkill.

Now, there needs to be a control plane, and the handoff between control and application planes needs to be well-enough defined to allow implementation changes in the future… And that’s probably it (until the system grows). The initial implementation doesn’t need to be built in the highest-performance programming language, use the fanciest storage technology when the users and tenant volumes don’t justify them in the control plane. In fact, early on, I would argue that simplicity and diagnosability are the values to cultivate above all.

As an example, for the entirety of my involvement in the SaaS rapid solution delivery platform at my last job, we were able to make do with human-readable files in a Git repo and some scripting as its main control plane. Heretical as that may sound, it was enough for current and projected scale. Was it fancy? No… But the desired state of the system was right there for the entire team to see when there were problems, plus the pull requests on the repo gave us relatively sophisticated review and historical auditing functionality without major up-front software development. I wouldn’t necessarily recommend the approach for everyone, but it does illustrate that you can often meet control plane your needs without spending a colossal amount of money or completely capitulating to a third party’s system (a.k.a., “do what the cloud provider tells us to do”).

The most dangerous word in software architecture

Wed, 16 Apr 2025 14:43:28 -0500

The most dangerous word in software architecture is “yes” — especially in its qualified form (“yes, but…” — where nobody will listen to the “but…” part).

I was chatting with colleagues about our biggest failures as software architects. We’ve built the wrong thing, failed to build things, built them buggy. We’ve made a lot of mistakes over the years, but the biggest ones — the product-killing ones are usually saying “yes” to something that might as well have opened a portal to Hell in the project. These were never frivolous “yes” answers. They were often tactical wins. But, what they did is establish a new expectation in the minds of stakeholders that made success significantly harder.

I know that it is popular to say that “just” is the most dangerous word, but those are tactical errors. The product-killing strategy errors are the agreements, taking on things that aren’t really feasible without a technological investment that you can’t afford.

I have done my share of them. They come in all shapes and sizes at the time, but they tend to take the form of doing something that you’d planned to eventually do, but moved up to right away. The problem with saying “yes” to such requests is that they often appear like small things to the outside world, yet doing them properly would require the product implementation to mature significantly. What happens then is usually one of two things:

The feature apparently working removes the pressure to make the required technological investment. Why move a mountain when “it already works” (the “but…” part in “yes, but…”). By the time the limitations are apparent, the needed investment has grown due to technical debt and the appetite to make it has passed.
The premature availability of a niche feature completely changes the perceived intent of the project. Gone are the original intentions and vision, all that is front and center is this niche feature. Sure, there are success stories of such distractions becoming successes in their own right (e.g., Twitter) but, more often than not, the new thing is not a world-changing product — it is a shiny distraction that ultimately kills itself and its parent in an unintentional murder-suicide.

I suppose that you could also view this post as a warning about the perils of careless product roadmap management — which is true — but the momentous roadmap change doesn’t necessarily look like one. It looks like an innocent, constructive question. A question whose consequences only become apparent much later.

So, you want to do SaaS? Read this book chapter first.

Mon, 14 Apr 2025 11:52:12 -0500

It feels weird to endorse a book just on its first chapter but, in terms of ROI, this could be the most important chapter in the book. The introductory first chapter in Building Multi-Tenant SaaS Architectures by Tod Golding should be a must-read for everybody involved in a software as a service (SaaS) business. It should also be a must-read for those who don’t want a SaaS business model. The fundamental thing with software as a service is that it is a business model supported by technology. The two go hand in hand and you can’t have one without the other.

So much pain and failure comes from the inability to recognize or admit that combination. Obviously, in the case of large companies, that does not mean that the entire business and technology foundation of the company needs to be that way – you can (with difficulty, I will admit) have a clearly carved out SaaS business within a company that is fundamentally not a SaaS company. The key thing is that within the boundaries of the SaaS endeavor, business and technology need to be aligned on the model.

I think it is also really valuable for readers whose conclusion after reading the chapter is “I can’t / don’t want to do that!” The recognition that the union of all the prerequisites is unacceptable to you is valuable. Pursuing the benefits of an unattainable model does nobody any good. The relative cost of $80 (the book’s current list price) and the time to read 22 pages vs. embarking on a doomed software project should be a no-brainer.

I could (and probably will) write at length about how the SaaS model can be applied as part of an overall organization that, as a whole, doesn’t meet the prerequisites for a SaaS business but the first and most important step is recognizing the reality of the business model and the commitment it requires.

5 more mistakes people make when building SaaS software

Wed, 09 Apr 2025 16:35:03 -0500

As a follow-up to my comments on the InfoQ “QCon London: Mistakes People Make Building SaaS Software” post, I thought I would add my list of five additional common mistakes that I’ve seen SaaS product teams make.

Mismatched revenue vs. cost scaling

This is mostly a business failure, but with significant engineering input: when offering SaaS that has a variable usage-based component that implies non-trivial costs, forgetting to include those costs in the product pricing (or forgetting to include limits to prevent abuse – see the next mistake) can end up being catastrophic. Before you say “yeah, that’s just a business problem”, I will point out that engineering needs to discover and report these costs. The business is not going to magically know what costs money and what doesn’t.

As an example, I once worked with a team whose costs ended up dominated by their logging provider: the provider charged by log volume, logs were proportional to the data ingested / processed by the system, but the price was scaled according to a lightly correlated variable that matched light usage, not real-world usage. By the time they realized their mistake, they had to endure expensive losses while working frantically to reduce their logging provider usage. The expensive things aren’t necessarily what you expected up front.

Not having limits

When I say “limits”, I mean limits on every part of the system – rates, volumes, backups, anything that can grow and costs money. Virtually everybody remembers to use an API rate limiter, but limiting all other variable aspects is equally important. This seems like technical debt that can be kicked down the road, but my experience is that in any kind of data-intensive system, there will be another component that drives cost in an unexpected way. Setting those limits (even if they are high) and testing them is a really important part of making sure you aren’t surprised.

This isn’t in contradiction with the previous mistake (mismatching revenue vs. cost scaling): even if revenue scales perfectly with usage, there is a point where customers will have sticker shock and rebel. Having a conversation about limits and making sure that they are on board with the implications of their usage is preferable to irritated customers refusing to pay.

Misjudging your organization’s willingness to iterate

This one goes both ways, depending on the kind of organization. One key advantage of the SaaS model is that you can be lean – deliver incrementally and adapt as you learn. The real danger lies when you rely on this but are in an inflexible organization. If you can’t continuously choose between adding features or iterating over past implementation based on feedback (especially operational feedback), you end up becoming trapped in an increasingly unproductive cycle of trying to add features to an inadequate base. The inadequacy can take many forms (quality issues, decisions that turned out to be incorrect, etc.), but they ultimately present as a lack of productivity that only gets worse over time.

The lesser form of this is to overshoot the completion of features before making them available, slowing down the rate at which you gather feedback and increasing waste when you do incorporate it. Unpalatable as it may seem, applying the perfect definition of “done” before anybody can see new work is also a mistake that gets projects cancelled for lack of progress. The trick is finding the level of iteration and leanness that the organization can tolerate.

Undershooting operational quality

The key word here is “operational” – most software development organizations have a pretty clear set of quality acceptance criteria for software on its own, the problem is when it actually is deployed. The ability to detect and resolve issues when the software is in operation is somewhat of an afterthought: the traditional definition of “done” tends to be along the lines of “we tested it on the bench with no errors”, not “we exposed it to the whims of the customer base, figured out what they broke and fixed it”. In the more extreme cases, you may be blind to the problems experienced by customers. That is not a good place to be. You want to know that there are problems as soon as possible, preferably fixing them before customers even have time to create a support ticket.

If you don’t have too many operational issues or they take too long to resolve (corollary: if you don’t feed back changes into the system itself based on your operational findings), all parties involved will be unhappy. Customers will be mad and look for alternatives, the development team will be demoralized and checked out. The #1 job of a SaaS software development team is keeping the system running smoothly (that is what customers pay for, after all), writing new features is a distant #2. Only when you’ve gotten good at operations and operational quality will you have the time to do fun things like develop new features.

Organizations that split development and operations thinking that this allows them to keep doing the fun stuff while others deal with fires only defer and magnify the consequences of this mistake.

Not having a proper tenant decommissioning plan

It’s nice to think of tenant onboarding, keeping them running, etc. but nothing lasts forever. Customers will eventually leave. If you can’t properly decommission their resources and destroy their data as agreed, you’ll be in a world of hurt: customers (especially in a B2B context) have real expectations of data destruction on termination (potentially delayed by some period of time) and get really mad if they find that their data hasn’t been destroyed as expected. Similarly, if you keep paying for resources used by departed customers, those stragglers will eat into your profits until the end of time.

It’s easy to think of this as a problem for later. The mistake is when “later” becomes “never.”

Commentary: "Mistakes People Make Building SaaS Software"

Wed, 09 Apr 2025 13:47:19 -0500

I was amused to see a clickbait-y article pop in my RSS feeds from InfoQ: “QCon London: Mistakes People Make Building SaaS Software”. While it is hard to really judge the substance of the talk from this article, its “top ten mistakes” list is a pretty good conversation starter for people who actually are thinking about getting into a SaaS business.

The list is not perfect and probably isn’t exactly how I’d break things down, but it is nice to have a starting point.

Let’s go over some of these:

Building something users don’t want

Delivering unique features or different versions to tenants

Listening to consultants instead of understanding your customer base

I think these are actually facets of the most fundamental problem getting into this business:

what kind of organization are you?
who are your customers?

I’ve seen this firsthand: years spent in a large organization that was at its core product sales-focused trying to get into a service delivery business was an incredibly frustrating experience. No amount of introspection at all levels of management could ever change the leopard’s spots, so to speak. Large organizations have a culture and a way of doing things that won’t reinvent itself, regardless of how hard they try. This culture often dictates what kinds of customers they are willing to pursue. So, the first question is incredibly important.

Of course, the larger the organization, the more likely it is to engage consultants, with predictably mixed results at best. In addition to the poor customer fit the list alludes to, the other pitfall of engaging consultants without tightly working with them is that they can be blind to feasibility challenges. Unless they are deeply knowledgeable about the customer’s domain, they probably won’t know the functional and, especially, non-functional challenges that can make a seemingly brilliant product idea into a software development black hole. Yes, working closely with consultants and customers is a good way to avoid those surprises… But, at the risk of being cynical, that’s not something that happens very often.

At the other end of the scale, small organizations have a lot of flexibility, leading to a lot of the temptation to make mistakes John Topper alludes to in his talk. Some customers and domains have requirements that directly lead to these mistakes, so walking away must always be an option. Being lean is being able to recognize early that some ideas aren’t worth it.

To be fair, you have to understand the actual requirements – not just their superficial expression – to make some of these decisions. For example, people that have been burned by software that changes dramatically under their feet (the redesign in iOS 7 being a great example) will be tempted to make dramatic, sweeping, stability requirements that are in direct conflict with other fundamental properties of the software (e.g., security, scalability). These can be often negotiated into a more benign form. If not, walk away. That is business that is fundamentally incompatible with SaaS.

Not building tenancy concepts into application components

Agreed. Since the SaaS business model is predicated on scale, minimizing the marginal cost of adding tenants and users while meeting their isolation and performance requirements is fundamental to profitability. Having lived through the debacle of “let’s start with a single-tenant system, we can fix it later”, I can tell you that virtually no amount of money for that first sale will ever make up for the broken business model it entrenched.

Not calculating baseline per-tenant costs and testing this with the market

Agreed. But, to be fair, calculating costs can be difficult if you have no experience in the domain and don’t know what to expect. It’s relatively easy to calculate worst-case per-tenant costs (in the degenerate case where you only have a single customer), but the real trick (from a business perspective) is to know a realistic number of tenants to amortize base costs for. An additional (technical) challenge is to find the non-linearities in resource costs and how they map to business concepts – e.g., when do you need to go from N to N+1 database instances? How many tenants, users, data volume units does that map to?

The other baseline and marginal costs to keep in mind are the support and operations cost. It’s easy to fixate on the infrastructure costs, but the more specialized the domain, the higher the risk that customers will expect humans to support them.

Not automating tenant provisioning

Agreed. That was one of the things we worked very hard on in my last project for SLB. Even when transaction costs are overwhelmingly non-technical, being able to provision tenants quickly and cheaply is a game changer.

Deploying your software into your customers' AWS accounts

Agreed. I’ve seen this more often with Microsoft Azure, where the “Azure credits” included in other IT purchases from Microsoft incentivize this kind of behavior. It isn’t just a trap from the perspective mentioned in the article (it’s shrink-wrap software, not SaaS), it is also an IP protection and service level agreement perspective: the customer has full administrative control over your software, can see exactly how it is made and can prevent it from working properly any time they want.

In common with on-premise distribution, there is also licensing peril with doing this: arguably, this is “distribution” from the perspective of software licenses (e.g., the GPL), which triggers a number of obligations not present when just delivering a SaaS service using the software package. Also, commercial third-party dependencies may incur unexpected additional costs when the target infrastructure doesn’t belong to you.

Building your solution to be multi-cloud or cloud-agnostic

With the caveat that there are certain domains where being able to use multiple cloud providers is a key requirement (e.g., for data residency in regions with limited cloud provider coverage), agreed. The cost of building a product that is cloud-portable is huge. Most of this cost is opportunity cost: not being able to fully use the capabilities of your chosen provider because there is no equivalent elsewhere. The more direct cost of building compatibility layers, etc. is also significant but it is fairly easy to quantify. We are far less good at quantifying the features we could build if we’d just used the perfect platform service from our single cloud provider, instead of doing the extra infrastructure work to support other providers and make up for the lowest common denominator.

Now, there are businesses where cloud-agnosticism is a key feature. That is just an incredibly expensive feature.

Deploying your solution on-premise for customers

Agreed. Not that there aren’t valid business models that involve on-premise software delivery, but they are virtually always incompatible with a SaaS business.

Not thinking sufficiently about backups and disaster recovery

Operations, backups and disaster recovery are all fundamental aspects of running a SaaS business. You have to make disaster recovery your religion to be anywhere close to good at it.

Not classifying data assets

While this is probably only intended from the perspective of compliance (e.g., GDPR) and threat modeling (based on Jon Topper’s other presentations – e.g., this), data classification is an interesting area that I’ve found to be pretty fluid over the years. Without a doubt, you have to classify your data to meet compliance and threat modeling requirements. How finely you classify the data beyond indentifying PII and security-critical information is a choice that needs to be taken carefully. It can easily be a deep rabbit hole with very little tangible reward.

Ten years ago, we were trying to model various facets of data in our systems to cover a wide range of concerns:

country of origin
country of modification
privacy
security criticality
allowed uses (model training, etc.)

And so forth…

The temptation was to tie data classification and data entitlements because this appeared to be the silver bullet for various legal requirements – which was hard to do and never worked well. In theory, you can construct your system out of classification-aware building blocks that apply an algebra of classification combinations to automatically classify outputs… In practice, it invariably falls apart and you realize that a less ambitious solution works just fine.

So, I think that I would rephrase the mistake thus: “not thinking about data classification and not classifying data where necessary”. The first step is to decide to what degree classification adds value (e.g., if it is a legal requirement, it is a no-brainer) before baking it into the system and development processes.

"We are not Google"

Mon, 07 Apr 2025 14:09:34 -0500

The most obvious, but also disregarded, piece of advice I used to give my colleagues is: “we are not Google”. By that I mean two things: that we were not operating at the scale of Google and, organizationally, we had nothing in common with Google engineering.

As early partners of Google Cloud, we were exposed to, and influenced by, a lot of their practices – which, to be clear, is often a good thing. I am a proponent of SRE, progressive delivery and, more broadly, DORA. What catches outsiders like us out is that these practices need to be tuned to the reality of our own organization.

The first problem is scale. The statistical underpinnings of these practices assume a large uniform set of users – two facts (large, uniform) that often aren’t true in the world of Enterprise and B2B software. When you have a billion users using your system globally, you can rely on statistics to give you a clear and timely picture. When there are far fewer users, when they are relying on your system to work during very specific business hours, when each tenant has its own SLA, then you have to adapt!

Charity Majors has spoken of her experience at Parse where the overall health of the platform would hide the fact that one specific tenant could be experiencing a major outage. This is a problem that we’d see over and over. It is very hard, tending towards impossible, to build a multi-tenant system that is resilient to localized problems without degradation for the tenant: the infrastructure and development time to do so are rarely affordable and there is always going to be something that you didn’t think of. So, if you have made non-functional promises to individual tenants, you also have to measure and react at tenant scale, not just globally. This is hard and, to some extent, anathema to Google’s concept of SRE.

The other multifaceted problem is organization: very few companies have the depth of engineering that hyperscale tech companies do. Those are organizations that understand the value of in-house expertise and can afford it. Obviously, small companies can’t afford it. But, even companies that are big enough to afford experts also need to recognize the value of expertise and be organized in such a way to make the it available to those who need it. When you’re a “normie” – potentially very successful – company, you can’t rely on there being someone who can tackle deep, extremely challenging, problems in esoteric areas. You may have deep knowledge in some domain (hopefully, your business domain!) but there won’t be a deep bench of experts to call in when some technology misbehaves in novel ways.

The two problems compound each other: not only are smaller organizations not set up to tackle deep technical problems when they occur, they are unlikely to have the institutional memory to recognize them when they occur – nobody will have seen even all the mildly esoteric problems before, never mind know how to fix them. Another way of putting this is that you should avoid solutions that you aren’t able to diagnose and debug within your team – there won’t be an expert that the company can pull out of a hat to fix them for you when things go badly wrong. This is not an endorsement of technological stagnation – far from it! – but rather the observation that advances in technology need to come with commensurate advances in one’s ability to operate and debug them.

I guess I should write about stuff, now...

Mon, 24 Mar 2025 15:31:26 -0500

Now that I have more time on my hands and no longer have to follow my former employer’s social media policy (which, admittedly, had improved over the years from the original Omertà-like policy), I can write more freely about the things I have learned over the years. Funnily enough, that’s when I saw this post implying the pointlessness of the exercise. Sure, he’s not saying that I shouldn’t speak – the implication is more that nobody should listen…

Well, I suppose that this is mostly aimed at “how to advance your career” advice, not “how to become a better software engineer or architect” advice. Yeah, I’m not going to write about that. Also, with time, the purely technical details will lose relevance – for example, there is very little topical relevance to the code of the automated hematology classification code I worked on during my internship at Technicon Instruments in the late 1980s (rewriting PL/M into C just isn’t a hot problem these days). However, even going that far back, there are constant threads related to people that recur today.

We may think that software engineering is just a technology problem, but it is first and foremost a people problem. Some of the things I saw in Tarrytown nearly 40 years ago still happen across the software industry today. Tools and processes are better, people are better educated, have access to a wealth of information we didn’t have at the time, but we’re still working with the same hardwired constraints of the human mind.

Finally, the fact that we – on the whole – know more about writing software doesn’t mean that we apply that knowledge uniformly. E.g., for all dependency layering within non-trivial programs is a problem we’ve tackled for decades, spawning generations of tools to help us (or, to “solve” the problem, if you listen to their marketing), it still is something that even experienced developers get wrong on a fairly frequent basis.

Fri, 21 Feb 2025 09:16:38 -0500

It turns out that having a price watch on the Canon refurbished store is an expensive idea… Who knew?

AI utility

Mon, 13 Jan 2025 23:08:19 -0500

I really like this quote by Alex Komoroske (via Simon Willison). It may be confirmation bias, but I find that it is a good way to think about the utility of AI, whether plain-old “machine learning” or of the current hyped LLM variety:

LLMs shouldn’t help you do less thinking, they should help you do more thinking.

They give you higher leverage.

Will that cause you to be satisfied with doing less, or driven to do more?

Of course, humanity is likely to take the wrong path here – we seem to never miss an opportunity to switch our brains off – but we can choose to do more with these tools. The temptation, whether by average people or businesses, to do less for (nearly) free is the road to Idiocracy.

Evaluating Peakto for picture library management

Sun, 12 Jan 2025 23:48:27 -0500

While my overall picture library isn’t huge, it is split between raw files edited in DxO PhotoLab, Apple Photos and a directory structure of legacy JPEGs taken with various digital cameras over the last 25 years. So, finding pictures or specific subjects is a bit of a challenge.

This is not a full review – I was intrigued enough by Peakto to do a free trial (which, for some reason, they make a lot harder than it should be). So, here are my findings after a couple of days…

First, the good:

The search capabilities are very nice, especially considering that it is searching multiple distinct catalogs or file systems. I don’t quite yet understand how much it derives from image analysis and how much keyword tags contribute to the search, but I do enjoy it.
Performance (ingestion, analysis, searching, etc.) is generally very good. My M1 Pro MacBook Pro is pretty fast, but it also isn’t the hottest machine learning hardware on the market, either.
The face recognition is impressive – it will identify faces even in the background of pictures and is generally fast.
Once you get used to the UI paradigm, it is quite productive.
It has some interesting support for video, with automated audio transcription, but I haven’t really played much with it.

Now, the bad:

The onboarding experience could be improved. A lot. I wish I had known to start by playing with the demo content then removing it (there is a setting for that) before trying to ingest my own pictures. The ingestion paradigm for the various different sources varies per source kind, so my own assumptions led me astray for a while.
There are some rather annoying UI bugs – none fatal – that detract from the experience. The most annoying is the way the keyword tag editor has keyboard focus problems that make typing tags a lot harder than it should be. The grid view refreshes itself needlessly when applying edits in the background. It is by no means bad, but the niggles are annoying.
The AI “analysis” feature seems questionable. Maybe I’m missing the point, but its aesthetic and technical rankings seem poorly correlated with my taste.
Sometimes, the DxO PhotoLab support seems to fail to pick up the post-edit thumbnail from PhotoLab. When the editing is significant, that can be quite frustrating.

Some things that could be improved:

The keyword tagging support is relatively limited. I didn’t see a way to edit hierarchical keywords, tags are case-sensitive but the application insists on one casing convention even when the existing tags in the library follow another and, finally, there is no global management of tags.
The face recognition is just that – it recognizes faces but does not let you tell the application where it missed some. There are some scenarios (rotation) where it is less effective at detecting faces and it would be really nice to tell it what it missed.
Related to the above, face recognition and keyword tagging are distinct features that don’t link to each other. This is annoying when the underlying application (e.g., DxO PhotoLab) understands tags, but not faces – being able to create tags based on faces would help when searching in PhotoLab.
Similarly AI-generated tags and tags manually associated with the images live in parallel worlds. You can use both, but combining them is clumsy.
Scrolling performance in the grid view could be smoother – especially when scrolling slowly (thus, the next pictures to show are known in advance).
As far as I can tell, some trivial image file metadata edits like rotation are missing. Working with directories of random legacy digital camera files is needlessly painful when you can’t even rotate the image orientation without jumping into another application.
Using YouTube as the onboarding documentation is weird. The videos are nice enough, but they aren’t that numerous or slickly produced to justify the emphasis.
Burying the free trial options so that people start a subscription to try the software may improve the conversion rate, but it sure isn’t a welcoming sign or a sign of confidence. As far as I can tell, you can try the software without starting a subscription, they just obfuscate it on their website. To be honest, I can’t tell whether it is a dark pattern of just bad web design, but it did make me think twice before trying Peakto.

Overall, I like the application and I find it to be a good step towards better management of my picture library. It’s pretty obvious that it is a niche product by a small company, so I understand that the polish and pricing can’t be as good as for a mass-market product, but this gets a job done that the big players aren’t tackling. So, Peakto definitely has a place.

In the context of DxO PhotoLab, I would rather that DxO put all their effort into the photo editing features if I could use Peakto for library metadata management and searching. There could be a pretty good synergy there.

"Write better code"

Fri, 03 Jan 2025 15:36:34 -0500

It’s pretty funny that all you apparently need to do to get better output from LLMs is to ask them to do better. Maybe I need to ask the JetBrains AI plugin to “write better code”. Of course, there is a gap between writing a simple program from scratch and writing contextually and stylistically consistent new features in an existing code base… But maybe it is worth a try?

AI-generated documentation

Thu, 19 Dec 2024 08:22:18 -0500

To clarify my snarky “junior developer” comment about code documentation in yesterday’s post about the JetBrains AI plugin, good documentation describes the intent of the code, it isn’t a blow-by-blow description of what the code does. While the ideal of literate programming is not practical for most software, it remains somewhat of a north star – you want to document what is not in the code, the rationale. So many beginners in this profession miss that point.

This is where the AI plugin has a hard time. It can only infer intent from the code as it is written. It doesn’t know what you had in your head while writing the code. It can’t tell whether this was a simple idea with a convoluted expression in code, for example. To be fair, it still is a starting point – sometimes, the required incremental description is small… Still, it falls far short of something good out of the box.

I fear that the automated descriptive documentation like this is only going to reinforce the misconception that “the code is the documentation” because this kind of documentation doesn’t add much beyond reading the code.n

Spending time with JetBrains AI

Thu, 19 Dec 2024 00:01:35 -0500

When it comes to the sci-fi trope of humanity ending at the hand of the machines, I’m in the camp of “The Creator” rather than “The Terminator” – it’ll be a software bug that does us in rather than machines achieving sentience and judging us unfit. Which brings me to… the glut of “AI” coding assistants that are so hyped these days.

With some time off for the holidays, I had an opportunity to update some of my open-source projects and try out the JetBrains AI plugin as I did it. The results are interesting…

On one hand, virtually none of the major features work perfectly – every time the plugin generated refactorings or unit tests, they were somewhat off, the documentation generation was at the most junior developer level, but I think that it would be wrong to call it useless. In fact, it is well worth the $100 yearly cost they charge for this thing because that “wrong” code it generates doesn’t work as-is, but it also isn’t that far off from what was needed. So, at the end of the day, using the plugin to start menial tasks is a huge time saver. It just doesn’t reduce the time to zero.

I did find the fact that the underlying prompts all start with “you are a rock-star … developer” to be quite amusing. Obviously, we all have different definitions of “rock-star developer”, but the output looked more like the “booze and groupies” kind. At some point, we have to remember that this doesn’t (yet) understand anything about what it is doing. It is incredibly good at mimicry and does work a lot faster than any human could. It is only a “10x developer” in that it is 10x faster, not 10x better. I think that there is still a large gap between where this is going and the humans that are good at this craft. I’m sure that the gap will narrow, but the intent to fully automate at any cost seems unjustified.

Which brings me back to the original comment about AI: we’re at a stage where these assistants are productivity multipliers for skilled workers. Just like computers have been in various fields for decades now. But that isn’t apparently good enough for the kool-aid drinking hype mongers: nothing less than complete replacement of the human element is acceptable in their book. The economics of the workplace aren’t static, though – it’ll certainly devalue large parts of the this profession, but the runtime cost of AI models still isn’t zero. There is an equilibrium to be found. I just have no idea how unpleasant it’ll be, especially for new practitioners, and I have no idea how innovation will continue to happen when everything is a “remix machine”, but it’ll be interesting to watch… I guess.

Velcro strips are your friend

Sun, 29 Sep 2024 11:10:30 -0500

The Walksnail Goggle X Extension Board arrived just in time for me to try flying an analog tinywhoop quadcopter that (hopefully) will be easier for a beginner to manage in a small back yard. I don’t have much to add about it, especially now that “real” reviewers have had time to give their opinion, but one point I think that they consistently forget is that the double-sided adhesive pad supplied with the module makes packing the goggles a total pain and (I think) should be avoided.

Self-adhesive heavy-duty Velcro strips are an obvious win here: you can set up the extension board, VRX and antennas as a unit (I zip-tied my VRX to the module so that it doesn’t pull out in transport) and attach/detach the whole unit with the Velcro. It isn’t a perfect contact area because the curves on the top of the goggles, but the Velcro strips do their job just fine. Apart from that, as the reviewers say, the extension board works fine with a Foxeer Wildfire VRX and there was really nothing to complain about given the Goggle X as starting point (so, no analog DVR)). I’m not picky enough about latency to notice anything worth mentioning and picture quality is… Well, analog.

Obviously, this would not be my first choice if I were looking for an analog goggle “daily driver”, but as a way of adding on analog video to a primarily digital ecosystem, it really isn’t bad at all. We can quibble about whether Walksnail should have designed a more analog module friendly goggle, but this isn’t bad as a “slap it on for occasional use” kind of thing.

BetaFPV Meteor85 first flight hot takes

Wed, 04 Sep 2024 18:00:09 -0500

The unboxing and first flight experience with the BetaFPV Meteor85 (as a first-time quadcopter user) was… Something.

Because the Meteor85 flight controller has SPI ELRS on board, flashed with ELRX 2.x firmware, the first thing I had to do was to flash the latest Betaflight firmware on the controller. That’s exactly what everybody warns you not to do, but I need the latest Betaflight firmware to get ELRS 3.x support on the SPI receiver. That was a bit of an adventure as the web Betaflight Configurator doesn’t seem to be able to make the flight controller enter DFU mode – but the desktop Mac app. does! After figuring out that little mess, things more or less just worked after updating the firmware and setting the ELRS binding phrase.

I noticed that air mode is enabled by default from the factory and the switch settings to toggle between flight modes favor angle mode, so I remapped the switches a little, but it was more or less ready to fly once firmware was up to date. I took the time to update the Walksnail VTX firmware (damn, the BetaFPV and Walksnail USB dongles are tiny and fiddly!), bound it to the goggles (oh, yeah, that binding button is well-hidden under the drone’s canopy) and did a quick test flight in the back yard.

Hot take #1: the Liftoff Micro Drones simulator doesn’t do the real flight characteristics justice. It isn’t miles away, but the real drone is far more unstable and fussy than the simulator. Once in the air, I think that it is closer but the takeoff experience is definitely harder than the simulation. Lots of splats into concrete and plants, there.

Hot take #2: I need a bigger back yard! Well, I need to practice somewhere bigger and wide open because that “drunken sailor” first flight experience between oak trees and tall vegetation is not the best…

Hot take #3: This 85mm whoop is really easy to lose in vegetation! It doesn’t have a buzzer or LEDs, so it is surprissingly hard to find once it goes down.

Hot take #4: The Walksnail Avatar 1S VTX on the Meteor85 overheats really fast! Not a problem once you’re flying, but when you’re mostly crashing and searching for the downed drone in he bushes, it is a race against time to find the drone before it overheats and shuts down.

Getting the Walksnail Avatar Goggles X to work with a Mac

Fri, 23 Aug 2024 17:52:31 -0500

Getting the Walksnail Avatar Goggles X to work with a Mac is rather annoying, but not quite the impossibility Caddx claim it to be:

The first challenge is dealing with the recessed HDMI-in micro-HDMI connector. In the end, I ordered a cable with a plastic (rather than metal) connector body and shaved it with a knife. It’s ugly, but it fits fine.

Once plugged in, assuming you’ve figured out how to switch inputs [a long ↩ (return) button press], then comes the finicky dance of picking a display mode that the goggles are willing to display. 1920x1080 at 100Hz eventually works fine for me, but it took multiple tries to get there. Initially, I got a solid magenta screen, then noise before it deigned to work. Now, it seems to work consistently, but I have no idea what made it work the first time. As far as I can tell, the HDCP handshake issues that Caddx said were a blocker are resolved in the current firmware. I didn’t need to run through a HDMI splitter to make it work – a direct cable from my MacBook Pro’s HDMI port to the goggles was fine.

Now, once you have a picture, it is all weirdly magenta with messed up colors. That is because the goggles advertise both RGB and YCbCr color modes and the Mac chooses YCbCr, but the goggles don’t really work properly unless in RGB mode. This is apparently a pretty common external display problem that, thankfully, the BetterDisplay utility solves this problem.

Following the instructions, if you have a version of BetterDisplay greater or equal to 3.0.4, there is direct support to change the color mode in the BetterDisplay menu bar item. It’s drama-free and simple (although the goggles sometimes seem to get stuck in a mode and require unplugging and plugging back in for the change to happen).

Goggles X HDMI-in "gate"

Thu, 22 Aug 2024 17:15:23 -0500

Walksnail Avatar Goggles X update: as reviewers noted, the HDMI-in connector (a mini-HDMI connector, as noted) is recessed and 🤬 is it difficult to find a mini-HDMI cable to fit! It seems that I’m going to be buying a bunch of cables from Amazon until I can find one that has a skinny enough connector body or that I can cut into shape… Boy, is this annoying.