There is now strong evidence that good management practices matter for firms, and that these practices can be improved by consulting. However, two big issues in translating this evidence into policy are cost and sustainability. Individual one-on-one consulting can be expensive – a global consulting company that we partnered with in India provided services valued at $250,000 per firm, and even using local consultants in Colombia cost about $29,000 per firm. Moreover, even if these consultants succeed in improving management, there are many reasons to worry about fade-out. Boston Consulting Group estimates that two-thirds of corporate transformations fail, as staff turns over and priorities change. Once a shock arrives, perhaps management is like a new habit that reverts to the old.
Together with Leo Iacovone and Bill Maloney, we worked with the government of Colombia to test out a novel solution to the cost issue: rather than just one-on-one individual consulting, we would have consultants work with small groups of 3-8 firms at a time in a collaborative setting, where firms could collectively try to improve and learn both from the consultants as well as from one another’s efforts. We tested this with a sample of 159 SMEs in the auto parts sector, with a second treatment group given individual consulting, and a control group just receiving a diagnostic. This group-based model was one-third of the cost ($10,000 per firm), and we found that in the first year it improved management practices at least as much as individual consulting, and improved sales and profits (see this summary of the original paper).
Measuring impacts 8-10 years later
In a new paper, Leo, Bill and I return to see whether these impacts are sustained 8-10 years later. Tracking impacts over such a long period is difficult for several reasons. First, the willingness of SMEs to answer surveys is typically much lower than for microenterprises and households, and can be even lower as more time passes. Second, over time, some firms close down, and survival becomes an outcome of interest in its own right, but also something to deal with in both data collection and analysis. We were fortunate that Colombia has excellent administrative data, and were able to link our firms to the annual manufacturing survey (EAM), the social security data (PILA), as well as to export data and an innovation survey (EDIT) conducted by the government. We supplemented this by having IPA call to verify which firms had closed down, and with detailed case studies to provide qualitative insights.
Group consulting had large and sustained impacts
We use these administrative data to track firms through to 8-10 years post-consulting.
· Group-treated firms were more likely to survive, especially those that were initially poorly managed: group-treated firms are 11-13 percentage points more likely to survive through to 2024 compared to the control group (for which 75% of firms were still operating). In the control group we see a positive association between baseline management levels and survival, with poorly managed firms being particularly likely to close. 62% of control firms in the bottom quartile of baseline management survive, whereas 92% of group-treated firms in this quartile survive.
· There are large long-term impacts on profits and sales: the long-term treatment effect is that annual sales increase by almost $1 million (55%), and annual profits by $382,000 (48%). Recall this is for an intervention that cost $10,000.
· Group-treated firms increase employment by 17 workers per firm or 29% conditional on survival. Our base estimates code firms that close as having zero workers. Our treatment impacts on levels then combine the extensive margin effect of keeping some firms open, with the intensive margin impact of operating firms having more workers. Control firms average 53 workers, and we see in Figure 1 the long-term impact is an increase of 17 workers. Using log employment, which conditions on operating, shows a 29% increase.
Figure 1: Group Consulting has long-term impacts on firm employment
We note that there is a missing market for this type of group consulting - firms may be worried about a lemons problem in peers if private consultants were to offer such a product. We therefore see a role for the government in solving this coordination problem, and encouraging firms in a sector to learn from one another as well as from the consulting experts in group consulting.
Typically, across a range of outcomes, we also have these two types of results which are a little harder to interpret:
· The point estimates for the group treatment tend to grow over time, but we cannot reject equality of treatments. Figure 1 shows this for employment, and we also see this for other outcomes. So we can’t necessarily say that impacts rise over time, but the results are at least not consistent with fade-out.
· Impacts for the more expensive individual consulting are typically positive, but smaller in magnitude than the group consulting (as in Figure 1). However, we can typically neither reject that these impacts are zero, nor that they are equal to the group treatment. The point estimate for the long-term impact on annual profits of $222,000 still greatly exceeds the cost of individual consulting, and we do find some significant impacts when we condition results on firm survival. But on a cost-effectiveness basis, the group consulting seems to dominate.
How did firms achieve sustained growth?
First, using the EDIT, we find management improvements appear to persist more in the group arm: 5-6 years after the intervention there is a 6 percentage point improvement relative to the control. Our qualitative case studies suggest the key factor was embedding these practices (or at least the ones firms felt useful) in their internal systems and using these as a foundation for future changes.
Second, some of the long-run growth appears to come through market expansion. Group-treated firms are about 10 percentage points more likely to be exporting a decade later. In the case studies, firms explained how this was particularly important to them when facing the COVID-19 pandemic and competition from Chinese imports. One owner remarked “I think that if we hadn’t started exporting, we would have shut down or gone bankrupt”. A manufacturer of rubber parts noted how the consulting had helped them think through their strategy for dealing with an oversaturated domestic market and heavy competition from Chinese mass producers. They focused on model-specific parts for which they could compete on quality, rather than price, and since Ecuador and Central America had similar car fleets, were able to extend this strategy to export in those markets.
Finally, the case studies suggest ways the group nature of the intervention had helped firms. The first was the usefulness of benchmarking. A manager in a cable manufacturer noted that “we looked at what was done in our company, and what other companies were doing – what we were doing that they weren’t, and vice versa”. Second, was the knowledge that they were not alone in dealing with the issue. A coupling manufacturer stated “I liked that I could see other companies facing the same problems”, and collective learning allowed them to reflect on common challenges.
Some broader lessons for long-run impact evaluations
The last 25-30 years have seen enormous growth in impact evaluations, with enough time passed to now revisit many of them and see whether impacts persist. But it needs to be both feasible to do so, and researchers need incentives to want to do this.
On the feasibility side, this is clearly much harder for phased-in randomizations, and for cases where the sample moves around a lot or is hard to otherwise re-interview. Administrative data can be incredibly beneficial here and lower the cost and risk of doing a long-term follow-up. But many countries do not have nearly the same administrative data as Colombia, and investment in building this data infrastructure will have a lot of spillover benefits for researchers. Yet administrative data will, at best, only tell part of the story. Typically, it gives us data on a much more limited set of variables than we might ask in our own surveys. This is particularly the case for trying to look at intermediate mechanisms. Using qualitative case studies for some deep dives into mechanisms and in understanding the patterns in quantitative data helps provide a lot more context and can help unpack what is going on behind the estimated treatment effects. A final point to note on feasibility is the need for funders to fund such work. This is particularly important for funding evaluations linked to World Bank operational projects, in which funding for the initial evaluation may come in part through the project, but then after the project is ended there is no such embedded process for long-term follow-ups. We are grateful to the Argidius Foundation for funding several long-term follow-ups of firm interventions through IPA.
Second, while long-term evidence is extremely important for understanding the cost-effectiveness and attractiveness to policy of interventions, the researcher incentives to conduct these follow-ups are sometimes a little less clear. This is particularly the case if the more limited set of variables available for long-term follow-up allow one to measure long-term impacts of the same outcomes already in a published paper, but then you lack a new theory or the ability to unpack the bundled intervention. Researchers can try to consider some additional outcomes that make sense over longer periods (for example, it takes time for survival and export effects to emerge), and they can make clear why looking at longer-term impacts is important. But they may still worry about whether this will be enough to be of interest to journals to publish. This is where newer short paper formats like the AER Insights length paper have really helped: a few key tables on the long-term results, coupled with a short discussion of what the authors can say about how to interpret them often works well (we are happy our paper is conditionally accepted there). In other cases I’d love the ability for there to be just 2-3 page “post-script papers” where the same journal that published the original study could add a single table or figure and a few paragraphs that say what happened after the study ended.
Join the Conversation