Decomposing Monoliths by Cleaning Rooms

Your engineers are telling you that the monolith needs to be split into several microservices. Work has slowed to a snails pace. It takes forever to get new features out. Changes often introduce bugs in areas of the code that weren’t even touched. It seems that any step forward is a step backward.

Have you ever cleaned a messy kid’s room? For me and my children it often starts like this, “Daddy I need help cleaning my room, it’s too hard!” The mess that they have made while playing has become too much to handle on their own. Monoliths are sometimes like that, built feature by feature until one day it’s too hard to move forward. Thankfully, there is a formula to clean up the mess.

Before I get stared with my child I have to determine if they are really ready to clean up. If they are too tired they will not be able to focus. Splitting a monolith involves the same inspection. Are you sure the engineers that built the monolith want to change? Microservices require an increased level of production readiness from the owning engineers. That muscle is going to take time to develop. This may require you to sell the change and potentially move or remove those opposed.

Also, it takes a certain type of engineer to do this work. I love this work. Give me a problem like this and I’ll go into my hobbit-hole for six months and return with the end product. Engineers that prefer greenfield work will be bored to tears (or quitting) by refactoring the same method call over 8,000 lines. It may be better to reserve them for maintenance and enhancement of the extracted services.

If the engineers are on-board there’s another hidden obstacle. When my children clean their room sometimes they try to push everything to the corners or pile it in the closet. The room looks somewhat clean but really the mess is just hidden better. Under the hood nothing has really changed. Are the engineers working on decomposition going to lead the system to the same state?

This hidden obstacle is very hard to avoid due to Conway’s law. Conway’s law states that organizations are constrained to produce systems that match the communication structure of the organization. I have witnessed this at multiple companies. Do you have a top-down organization? You will get a few god services that run the show. Do you have an office environment where one team can turn around and talk to another? You will get an tangled mess where services reach into other services to get work done. To fix this you may need to restructure teams and the building. Segregate teams and ensure they communicate through the proper channels.

Now it’s time to begin cleaning the room. When a mess is especially bad I help my kids by pushing every toy to the middle of the room. My son likes this because it creates a huge pile of toys. Sometimes I go as far as emptying every toy box onto the pile. This large pile is your monolith. Each function is a toy, individually distinguishable but together a large mess.

When the large pile is created the next step is to start defining sub-piles. I might know that there are Lincoln Logs, Legos, and toy cars in the pile. My son and I will create three piles and begin picking through the large pile for these items. Filtering the large pile is a lot of work and not all of the piles are known up front. We may discover that there should be a pile for toy trains as they are uncovered.

Decomposing a monolith is similar. You may have a general idea of the bounded contexts (sub-piles) within the code. As you start refactoring though you will discover more that need to be created. Some of these may even preempt work currently in progress. This can be very frustrating for those going through the process for the first time. They might expect that an architect, someone in charge, or even that they have it all mapped out. With a large monolith that’s practically impossible and if possible it is ill-advised.

Creating these bounded context cannot be done in a vacuum. It can be very tempting to put the engineer who knows the most about an area in charge. This can end in disaster. The engineer may say something like, “This process is great for an unknown system but is unnecessary in this case because I know it so well.” You may end up with microservices but they will be structured like the existing services. That would be like dividing the massive pile into smaller plies and placing each in a separate room. The term for that mess is a distributed monolith and is actually worse than a normal monolith. Toy trains are in each room now and it’s hard to play with them all at once. Distributed monoliths cause network traffic and costs to shoot through the roof.

To form a bounded context then you put a team together to start the discovery process. The team will interview domain experts which will help them to determine if they have a valid sub-pile or not. Those domain experts will be used throughout the decomposition process to validate the bounded context along the way. Have the team read Domain Driven Design before they start. It is difficult but imperative that this stage is done right.

Once the bounded context is created your engineers can begin the refactor work. This is like me and my son digging through the large pile to categorize toys into their appropriate sub-plies. This is where resolve comes in on your part. Project owners and managers will be frustrated or confused. Their feature work is even slower now. They may ask, “Why are we investing in the monolith when we’re just going to throw the work out?”

In a way they have a point. It would be better if the room could be picked up and things can be placed in order without creating the large pile. In some systems this is possible. Maybe you have some well defined contexts and just a few things are out of place. In other systems the monolith is too far gone. Those require the discovery and subdivision before splitting. In extreme cases it may feel like it would be faster just to rewrite everything. Burn the room down and start fresh. This is known as a big-bang rewrite and almost never works because you loose out on learnings of the past. Besides, the end goal is not to throw out work but to extract it.

On the other end of resolve you may have to slow down the engineers that want to get to the end state fast. I have seen this masked as “Getting to value.” They may want to skip some steps because it’s too much work right now. You may have to encourage some good engineering hygiene. The patterns and practices are extra work but in the long run create a more robust system.

Getting to value is a great mindset though. During the decomposition process the way to get to value fast is to work within the monolith. Build a well defined API using the bounded context discovery work. Then wire that well factored API up using the existing messy code underneath. This will make the engineers cringe but will prove out that the API is valuable and correct. The code underneath can then be straightened up to match the well defined API.

Once my son and I have created some tidy sub-piles then we begin moving them to boxes. When a box is well worn toys may spill from the holes so it is important to inspect them before use. Similarly, ensure the new APIs have well defined walls or seams. One API should not reach into another to change it’s state. This may look like the orders-api storing it’s data into an orders-data-api. Or an orchestration-api reaching into multiple APIs to “set things up.” This is more art than science and good looks to minimize the network traffic on each request to the system. Systems should act as sources of truth and work even if one component is down. Eventual consistency is key here.

Do you need to pause all feature work while the decomposition process is going on? No, but you do need communication. New feature work needs to go through the same discovery process as the existing work. Then when you are sure of what sub-pile the new work belongs you can either create the pile or add it to the existing work.

The largest killer of this whole process is lack of resolve. When my son begins playing with toys instead of sorting them I have to gently correct him so that we stay on focus. To you this might look like a priority shift. Perhaps one of the products is on fire and you need to shift resources to fight it. Don’t! The truth is this refactor work is likely a multi-year process for the first system. For every week that you disrupt a team you likely set them back two.

Another killer is rushing. When you hear an engineer say it’ll take six months they might be overconfident and it’ll really take a year. It will be tempting to put the engineer with the lowest estimation in charge. The decomposition process takes a long time, there is no way to speed it up.

A cousin to rushing is attempting to throw more people at the problem. If I only had more engineers this project would move faster! The truth is once the bounded contexts have been defined you may only need a single engineer to execute on the refactor. Adding more people just increases the lines of communication and slows the work down.

The truth is the time to pull systems out will decrease dramatically after each system is removed. The first may be a multi-year project but the second and third will be faster. This is why my son likes the large pile. The first few sub-piles take time but once the cruft is out of the way it gets faster to sort toys. The same thing happens with code. The first refactor is actually touches on the second and third. Each pass is a bit faster than the last.

So is it worth it in the end? A monolith in and of itself is not bad. I have seen monoliths scale businesses to 100M+ in revenue. I have also seen new products formed from the extracted APIs and new life breathed into old companies.

It can be worth it. I love seeing my son’s happy dance at the end of the process, “There is so much room to play daddy!” The questions you need to answer are; “How stuck are my engineers?” and “How much resolve do I have?” Once you know the answers to both you can begin the process.

Cannot read property ‘replace’ of undefined

If you get the following while setting up a new React/Babel/Webpack project you forgot to install all of the presets.


> webpack-dev-server --content-base client --inline --hot

/.../node_modules/webpack/lib/NormalModuleFactory.js:72
var elements = request.replace(/^-?!+/, "").replace(/!!+/g, "!").split("!");
^

TypeError: Cannot read property 'replace' of undefined
at /Users/.../node_modules/webpack/lib/NormalModuleFactory.js:72:26
at /Users/.../node_modules/webpack/lib/NormalModuleFactory.js:28:4
at /Users/.../node_modules/webpack/lib/NormalModuleFactory.js:159:3
at NormalModuleFactory.applyPluginsAsyncWaterfall (/Users/.../node_modules/tapable/lib/Tapable.js:75:69)
at NormalModuleFactory.create (/Users/.../node_modules/webpack/lib/NormalModuleFactory.js:144:8)
at /Users/.../node_modules/webpack/lib/Compilation.js:214:11
at /Users/.../node_modules/async/lib/async.js:181:20
at Object.async.forEachOf.async.eachOf (/Users/.../node_modules/async/lib/async.js:233:13)
at Object.async.forEach.async.each (/Users/.../node_modules/async/lib/async.js:209:22)
at Compilation.addModuleDependencies (/Users/.../node_modules/webpack/lib/Compilation.js:185:8)

Fix by installing the presets with yarn/npm:

yarn add babel-preset-es2015 --dev
yarn add babel-preset-react --dev

More than a “Hello World” in VBA

When I started at Bates Group, LLC one of my first assignments was to debug an Excel VBA macro.  Knowing nothing about the language I fought my way through the bug and fixed the macro.  After that I quickly decided to learn more about the language.   Since “Hello World” only gets me so far, I decided to do something a little tougher.  What better way to do that than to think back to my college assignments?

Back then one of the assignments I had was to write a random walk function.  Imagine standing next to a lamppost on the street.  From the lamppost you can take a step in one of four directions; North, South, East, or West.  You take a step in a random direction and then look at where you are.  From your new location you take another step in a random direction and you keep taking these random steps for a while.  Finally you stop and look up, how far away from the lamppost are you?

The following function does that only much faster than you or I could.  It takes 20,000 steps total and colors them along the way.  Every 2,000 steps it will change colors leaving a cool trail as it goes along.

Public Sub TakeAWalk()
    Workbooks("themikecom_vba_helloworld.xls").Activate
    ActiveWorkbook.Worksheets("Board").Select

    ' Where on the sheet should we start?
    ActiveSheet.Range("EE150").Select

    ' How many steps per turn should we take?
    STEPS_PER_TURN = 2000

    ' How many turns should we take?
    TURNS = 10

    For j = 3 To (TURNS + 3)
    For i = 0 To STEPS_PER_TURN

        ' Should we step east or west?
        randomX = Int(4 * Rnd)

        ' Should we step north or south?
        randomY = Int(4 * Rnd)

        ' Move west-east
        Select Case randomX
            Case 2 ' Move one step west
                If ActiveCell.Column < 1 Then ' Do not overstep the west border
                    ActiveCell.Offset(0, -1).Select
                End If

            ' Case 1 - Stay in the same spot

            Case 0 ' Move one step east
                If ActiveCell.Column <= 255 Then ' Do not overstep the east border                     ActiveCell.Offset(0, 1).Select                 End If         End Select         ' Move north-south         Select Case randomY             Case 2 ' Move one step north                 If ActiveCell.Row > 1 Then ' Do not overstep the north border
                    ActiveCell.Offset(-1, 0).Select
                End If

            ' Case 1 - Stay in the same spot

            Case 0 ' Move one step south
                If ActiveCell.Row <= 65535 Then ' Do not overstep the south border
                    ActiveCell.Offset(1, 0).Select
                End If
        End Select

        ' Leave a trail
        ActiveCell.Interior.ColorIndex = j

    Next i
    Next j
End Sub

Macro Flower ShotWith that done I wanted to add another function to learn how to create a menu.  I came up with the square flower.  This function will generate a square of random size with each section of the square filled with a different color.  This function taught me some tricks about looping in VBA, some ways are a lot faster than others.

Public Sub Flower()
    Workbooks("themikecom_vba_helloworld.xls").Activate
    ActiveWorkbook.Worksheets("Board").Select

    Dim start As Range
    Dim Length As Integer
    Dim Width As Integer
    Dim Color As Integer

    ' The starting point of the flower
    Set start = ActiveCell

    ' The maximum size of the flower
    size = Int(57 * Rnd)

    ' Ignore boundry errors for now
    On Error Resume Next

    For z = 0 To size
        ' Generate a random color for this row
        Color = Int((56 - 1 + 1) * Rnd + 1)

        ' Left side
        Range(start.Offset(0, 0), start.Offset(Length, 0)).Interior.ColorIndex = Color

        ' Bottom side
        Range(start.Offset(Length, 0), start.Offset(Length, Length)).Interior.ColorIndex = Color

        ' Upper side
        Range(start.Offset(0, 0), start.Offset(0, Width)).Interior.ColorIndex = Color

        ' Right side
        Range(start.Offset(0, Width), start.Offset(Width, Width)).Interior.ColorIndex = Color

        Set start = start.Offset(-1, -1)
        Length = Length + 2
        Width = Width + 2
    Next z

    On Error GoTo 0
End Sub

So what did I learn after all of this?  Mostly that I have a strong dislike for VBA.  It works well for small projects with small data sets.  However those small projects quickly expand into real programs which need to be maintained.  You are better off doing it right the first time instead of maintaining a large clunky macro.Excel Random Walk

Download the complete macro here. You will need to enable macros in your security settings to get them to work.  Once enabled, select “Random Walk” from the “theMike.com – Hello World VBA” menu.  This will start a random walk which will finish after a couple of seconds.  The “Square Flower” menu item will create a square flower under your cursor.

Starting a Windows Forms .NET Application

This week at work I have been redesigning the flow of our internal application. When Sally gets into work the first thing she does is launch this internal application. The application opens to a login form which disappears when she enters her credentials. A secondary screen then shows up asking which set of data she would like to work with. She chooses what she’s been assigned to work on and waits for the data to load in a third screen.

The code required to launch such an application is very straight forward. The login form needs to be displayed and when it is dismissed the secondary form can be shown. The third screen can be launched off of the second and will stay alive as long as the second is open.

Program.cs

///
/// Entry point for the test application
///
static class Program
{
    ///
    /// The main entry point for the application.
    ///
    [STAThread]
    static void Main()
    {
        Application.EnableVisualStyles();
        Application.SetCompatibleTextRenderingDefault(false);

        // Ask the user to login
        Login.LoginScreen login = new Login.LoginScreen();
        Application.Run(login);

        if (login.DialogResult == DialogResult.OK)
        {
            // Run the main application
            Application.Run(new NewWindowApp());
        }
    }
}

Take a look at lines 17 and 22 in the code sample from Program.cs.  The calls to Application.Run(Form) start a message loop on the current thread enabling the form to receive Windows messages.  Application.Run(Form) essentially says to Windows, “Here take this form and show it to the user.”  The form that is supplied can launch other child forms but when it dies so does the whole application.

My goal this week has been to make Sally’s job a little easier by removing the secondary screen and building its functionality into the third screen.  Now normally this would be as simple as replacing line ten with Application.Run(new ThirdForm());.  However that will not work in this case because the third form has a very useful “New Window” button.

Oh, it will work just fine for a while. Sally can click the button leaving her with two views of the application. However the message loop is only hooked up to the first form and if it is closed both forms will die off.

The solution to this problem is in an overload to the Application.Run method that takes an ApplicationContext property. In fact the documentation at MSDN shows a partial solution to my problem.

Instead of supplying Windows with a Form to display to the user I now supply a custom ApplicationContext object.  This custom context can manage the new window call and keep the application alive no mater which form is closed.

Context.cs

/// 
/// Does the work of firing off new application windows.
/// 
internal class Context : ApplicationContext
{
    // The number of windows currently open
    private int mWindowCount;

    /// 
    /// Initializes a new instance of the  class.
    /// 
    public Context()
    {
        this.NewWindow();
    }

    /// 
    /// Creates and shows a new window.
    /// 
    public void NewWindow()
    {
        NewWindow window = new NewWindow(this);

        window.FormClosed += new FormClosedEventHandler(window_FormClosed);

        ++this.mWindowCount;

        window.Show();
    }

    // Close out the application when all windows have been exited
    private void window_FormClosed(object sender, FormClosedEventArgs e)
    {
        NewWindow window = sender as NewWindow;            
        if (window != null)
        {
            window.FormClosed -= this.window_FormClosed;
        }

        --this.mWindowCount;

        if (this.mWindowCount <= 0)
        {
            this.ExitThread();
        }
    }
}

It's important to note that this object knows how many windows there are.  Also note that as each window is created by NewWindow() a listener is added to the FormClosed event.  These two elements allow the context to know when the last form is closed and therefore when to finally exit.

Full source code for the project.