AWS Takeaways

Recently, here at Hyla, my team and I pushed forward and got the entire trade in and processing platform in AWS. Along the way we made few application related improvements to leverage the benefits from the target environment. A recent LinkedIn post resulted in few folks asking me about the learning. Thought, I will provide a brief write up on our experience pushing through this change. Hoping that this helps others who are on a similar journey

#1. Change buy in – Moving to any cloud environment from legacy environment may not be a trivial activity. Hence, it is important that the change is not treated as a hobby project, has support from the senior leadership, has acceptance from business that it is an important strategic initiative and is strategically aligned with the business goals, has recognition from corporate finance on CapEx to OpEx shift. With the above in place, on the execution side, it is important to ensure that there is excellent project management team, commitment from the execution staff and, most importantly, good old fashioned grit to see things through.

#2: Leading change – The mechanics of pushing through the change boils down to (1) leading the change from the front (2) influencing the team to push through the change (3) issuing top down edict to execute the change. However, any change implementation comes with a degree of Fear, Uncertainty and Doubt (FUD). The leadership stamina to see through the change is directly proportional to intensity of FUD. Accordingly, appropriate strategy needs to be applied to push through the change. Leading from the front involves getting dirt under the nails. So if the leader has ability and bandwidth to embrace this strategy, the probability of success is relatively high. However, if the change execution is by influence, or if it is being pushed top down, then the leader(s) must ensure that team is adequately staffed and trained to successfully manage through the change.

#3. Architecture – While it sounds trite, it is important to pen down a high level and next level architectural diagrams that includes the low level details like VPC, Subnets for various tiers, ACL for the subnets, Security groups for EC2 including ports. It is important to keep in mind various services that may be needed as well – for example – services like SSH (22), SMTP (25), Tomcat (8080) etc to design the architecture. Using the architecture as blueprint, cloud formation or other scripting, needs to be written to build the infrastructure.

#4. Application State – When porting over legacy applications this is one area that mostly likely is going to cause a lot of heartburn. The underlying issue is what Martin Fowler calls as “Snowflake Server”. This is where folks needs to spend energy to decouple application state from the environment. One of the long pole happens to be property files. The best way to tackle this would be pivot to something like Cloud Config or Zookeeper or Consul. However, due to timelines pressure, it may be hard to pivot, and in those cases S3 could be leveraged to store the application state and configuration files.

#5. AWS Accounts – Before building anything it is important to think through account and hierarchies. One could design a fine grained hierarchy or stay coarse grained, and the final design needs to be driven by department or company objectives. In our case, we just needed four separate accounts for each environment – prod, uat, qa and dev. However, in larger organization it may be a wise idea to put deeper thoughts into account organization. This enables the ability to get billing information by account (it is also possible to get billing information using tags, and hence the reason to think through before hand as to how it needs to be set up)

#6. VPC and CIDR Ranges – It is equally important to put thoughts on segregating CIDR ranges based on environment and business domains. In our case, we had to go through few iterations to pick the right CIDR range for dev, qa, uat and prod (the few iteration could have been avoided if time was spent early on)

#7. Building up infrastructure – Building, by hand, through console is great for learning. However, folks need to invest time and energy to build up the infrastructure using CloudFormation or TerraForm or CloudFormation Template Generator (from Monsanto Engineering). In our case, we ended using CloudFormation (after a very brief evaluation of the other two products). Once the scripts are in place, it is important to start treating these scripts as code, specifically, infrastructure as code. This idea need to get ingrained as part of the software organization culture that infrastructure is no different from the rest of the code base. In our case, the cloud formation scripts are in Git and, going forward, changes to environment will get no different treatment than changes to code supporting our product suites.

#5. Service Limits – It is a good idea to be aware of what the limits are and make requests for adds ahead of time. It may not be ideal when an application under load, trying to scale up, hits the limits and breaks down. That may not yield optimal experience would it?

#6. Accessing EC2 – If set up right, only few (very few) will be needing SSH access to EC2. In fact, in a well automated state, even SSH access may not be required. One of the reasons developers need access to EC2 instances is to view application logs. The logs, however, can be piped to CloudWatch Logs and if the IAM is set up correctly, this should address the need for accessing the logs for debugging purposes. Another strategy would be send all the log data to ElasticSearch, which is actually the most ideal solution. This would not only enable enhanced search capabilities, but also opens up opportunities to perform log analytics through Kibana.

#7. Static IPs – In the cloud environment, there is limited need or no need for static IPs . However, this idea requires a little bit of getting used to, especially, when we are used to fixed IPs throughout our software life. In our case, only NAT Gateways have Elastic IPs. Pretty much every thing else in our environment have virtual IPs and almost all of them are private too. The SSH Bastions have public IP but are not static. So if the cloud formation that was used to build up the bastion were to be deleted and redone, the bastions will get new IPs. We felt that is OK given the fact that only few had access.

#8. Private IPs – Almost all of our IPs are private and none of them is visible to the outside world. The public IPs are for NAT Gateways, external facing ELBs and Bastions. One can access the private IPs only from the bastion. Initially, this process caused bit of pain because every time we needed to SSH to our EC2 resource we had to figure out what the IP was. This meant logging into the console to see what the private IP. This process required few more clicks than earlier. However, with automation leveraging AWS cli this problem is being aggressively tackled by our capable DevOps team.

#9. ASG – To scale we had set up CPU High and Low Alarms. Here too, it is a good idea to put some thought into what the high threshold and low threshold should look like. This one we learnt by experience. At one point our application servers were trashing pretty bad. In the middle of debugging, the server will just power off. The shutdown felt arbitrary with no apparent reasons. We went chasing our tail, thinking that the environment was “unstable”, suspecting something was wrong with the UserData part of the EC2. In the end, it turned out that the High CPU Alarm threshold was not right. The bar was too low, and when the application hit the low bar for the high threshold, ASG terminated the instance and replaced with new instance which then terminated promptly. Resetting the High CPU Alarms for Auto Scaling brought stability and relief.

#10. Tags – Putting thoughts into tags is extremely important. Tags are free form text and hence it is important to establish a solid naming convention for all the resources and diligently stick to it. This has potential to become run away and chaotic if not controlled from the get go.

#11. SSL Termination – Terminating SSL in ELB offloads the SSL overhead away from the webservers. In addition, AWS provides nicely packaged predefined policies for security which makes security a breeze (example turning off TLS V1.0 is a walk in the park)

#12. RDS – Going down this route takes away lot of freedom that comes with, say, setting up Postgres on EC2 (or MySQL on EC2). AWS retains the rights of true “superuser” and the admin user is limited to restricted set of privileges. For legacy application this is another area where people may have to spend time cleaning up. Another neat thing about RDS is that encrypting data at rest is a breeze. However, it might be a good idea to generate key from KMS and use it rather than use the default one.

#13. IAM Groups and Users – Time need to be put in to design and build out of IAM groups with appropriate set of permissions. The users can be assigned to the groups which gives better control over limiting permissions as well as achieving well thought out separation of responsibilities.

#14. Getting Help – The free support through AWS Forums is totally useless. Questions goes unanswered. Ponying up $ for support is well worth it (because of reasons mentioned in #15)

#15. Still Not Perfect – AWS is not yet perfect. For instance, during our production DB build out, Read Only Replica failed for unknown reason. It took multiple attempts with some help to AWS support to get rid of the zombie read only replica that sat in a limbo state for 12+ hours. During another time, we encountered an issue with the Cloudformation script. Specifically, we ran into situation where we were unable to delete a script because it relied on another script that was deleted successfully during an earlier time. The error message indicated that the script couldn’t be deleted because it used an export from the other script that was long gone (but managed to stick around behind the curtains in a phantom state).

#16. /var/log/cloud-init-output – During the build out phase, reviewing the output log in this location makes debugging UserData a breeze. The output clearly tells what went wrong.

#17. CodeDeploy woes. We used the “AutoScalingGroups” bit in the “AWS::CodeDeploy::DeploymentGroup”. However, every now and then, the ASG went into a weird state. To fix this state, meant we had to clean things up manually, which involved getting a list of ASG life cycle hooks and then identifying the one for CodeDeploy and then manually delete it using CLI. When this became a recurring pain, we switched over to Ec2TagFilters which made life a lot easier.

#18. CloudFormation – Keeping the scripts small and building one bit on top of another keeps the scripts organized, manageable and error free. We started with monolithic scripts with thousands and thousands of lines of code. We rapidly realized this was going to be problematic, and pivoted over to breaking it apart. So we built the core infrastructure (VPC,Internet Gateway, Nat Gateway, Route Tables, Routes etc), followed by web infrastructure (ELB, SG etc), webserver (ASG, Alarms, etc), appserver etc. We build up one after another using exports from the previous script.

#19. Lambda – We used Lambdas to execute custom logic in CodePipeline. The custom logic involved executing shell scripts in EC2 instances and moving files from one S3 bucket to another. The shell script were executed from CodePipeline through Lambda and SSM (it is bit complex that we like it). In addition, we utilized Lambda to send EC2, ASG and RDS Alarms and CodePipeline Approvals to HipChat Room. We think Lambda’s provides solid potential in AWS environment to automate many manual tasks.

#20. AWS Lock In – AWS provides amazing set of tools (CLI) and SDK (Java, Python etc) that makes automation a breeze. In addition, AWS is also starting to offer neat solutions for Code Build, Deploy etc that seamlessly inter operates with other AWS services and technology stacks. Leveraging more and more of these, means we are tightly coupling the applications and processes to the “virtual” environment. Such coupling means, moving to another cloud provider like Azure or GCP in the future will be lot harder to execute. So before digging deeper, it is important to evaluate the long term cloud strategy and have a crisp view on the path being taken. (same logic holds true for reserved instances)

Note: There were areas that we just couldn’t get to prior to production push but plan to tackle soon (1) Evaluate the ELB health checks instead of EC2 to make auto scaling determination (2) Evaluate federation option in lieu of of clustering to avoid network partition issue which seem to happen every now and then (3) Evaluate custom metrics instead of the free ones (4) Use Stacked Set for Cloudformation (5) CloudTrail for Audit (6) Granular Billing Alerts (7) Evaluate the use of reserved instances to save some more money (8) Explore Cloudian to reduce the cost even further

Microservices and organizational barriers

Off late more and more I hear about traditional IT organizations wanting to implement microservices architectural pattern. The underlying reasoning is that some of the tech leaders in these organization view “microservices” as a technical solution that will enable them to achieve velocity in the “market” (i.e deliver faster, better, cheaper). However I see that some of these folk fail to see that microservices is an architectural style like Service Oriented Architecture rather than an tool or a product or a solution that could be purchased, installed and configured. Martin fowler in this excellent article talked about technical & non technical aspects related to micro service architecture. In this post I would like to discuss some of the underlying organizational structure and implications.

There are some key considerations in any traditionally structured organization wanting to go a microservices centric model. Yes it true that microservices centric organizations are able to roll out products at amazing velocity (example. Amazon). But then there are two key considerations to building an effective microservices centric architecture and they are (1) organizational culture and (2) organizational architecture. In this post I will tackle the org architecture and not culture. A _typical_ IT organization is structured like this
OrgStructure
Some organizations are structured along technology lines, others are structured along business process line or some times combinations of both. For instance, an organization structured around technology lines will have one leader incharge of ERP systems (like SAP) and other leader incharge of .NET based web applications and other leader incharge of all Java based applications. In a business process centric organization, for instance,as in a Mortgage Banking technology group, one leader will be heading up all mortgage originations related application, another leader would be incharge of all servicing applications etc. In such a set up, implementing a microservices centric application architecture will be extremely complicated or even impossible. In this set up, monolightic application is a natural way of getting things done. The monolithic application has targetted purpose and the applications are structured to along the lines of hiearcical organization.

One of the reasons the IT organizations are organized the way they are… is to achieve cost efficiencies! period. A typical IT organization is a supporting group (excludes software product firms) and delivers capabilities to _support_ the business. IT is a cost center and hence there is ALWAYs the constant pressure to reduce the OpEx. One of the ways to reduce expense is by establishing hiearchical centralization to help achieve scale Human Resource efficiencies (spread a developer across multiple projects). In the last couple of decades offshoring and outsourcing has been another tool to reduce expenses. While some organization claim they are in India or elsewhere for talent arbitrage, most of the traditional technology shop leverage offshore as cost arbitrage opportunity with imposed structure supporting monolithic legacy applications.

In constrast a service oriented organization is more decentralized. In that model, what we will have is a set of services that provides certain functionality. To make it concrete, lets take mortgage as an example. Some of the possible services are… Modification Service, Underwriting Service, Risk Service (default risk, liquidity risk etc), Customer Service etc. From organization perspective, the service is an atomic unit supported by a team that builds, maintains and runs the service. The team is accountable for the service and is on the hook to keep up with changes. In the case of Modification Service, if there is a HAMP program, the team enhance the service to incorporate HAMP related logic in the service offering as manadated by regulatory bodies. The app developers in the service oriented world will expose smart and rich end points (and not dumb ones) that could be consumed by any application (web, mobile, voice, gesture). In addition, with this set up, the service now can be scaled up and down independently. A real life use case – during the mortage crisis, with the number of defaults increasing rapidly many many mortgage companies struggled to keep up with the volumes both on the technology side as well as on the organizational side (ex. call center staffing). Even though firms were rapidly able to hire people to service the defaulting customers, technology pretty much struggled for a long time because of inability to scale during times of crisis. The companies had to scale multiple monolightic applications (there by increasing cost) just to keep the lights on. With service oriented approach, it would have been possible to scale up the critical default related services rather than a shot gun approach of scaling up everything.

In addition, with service oriented set up, companies need to fundamentally shift thinking on how projects are conceputalized and executed. For instance, today, many companies still think along the lines of creating a business case for a capitalizable capital projects with distinct start and end dates. One the business case gets approved, the capex projects builds upon existing application(s) by layering in more features and functions. While model works, it doesnt align with service oriented thinking. Companies needs to fundamentally shift to into a product centric thinking instead of project centric thinking. For instance, a mortage company could think of Modification Service, Under Writing service as a product. Features need to be built into the product so that it can support the business needs. It will require a forward thinking product strategist and a product owner who can define the lifecycle & roadmap for the product. Now if a company figures out a killer Under Writing service it could now think of monitizing by exposing the service to other morgage companies (possibly after fully vetting out any legal implications)

Once such services are in place, then creating composite application that leverage the services becomes _relatively_ simple. An user interface centric application (read mobile or web) or Non UI application (voice, gesture) can access the services and provide the desired outcomes. This model is extremely desirable especially now when the desire for achieving rapid velocity to rollout differentiated products and services is more than any other time. Traditional companies steeped in motholithic application are unable to respond fast enough. The business leaders perinneal gripe is that IT is not fast enough. However with focus on cost supported by rigid org structure & traditional mindset, CIO have miles and miles of ground to cover before they can be nimble like the way their business leaders wants them to be. The business leaders on the other hand, need to stop looking at technology as a cost center, and need to make strategic investments. However in the interim the chasm created by what is possible versus what is current is giving opportunities for the new, nimble service oriented upstart to come in and disrupt the established players (example Honest Dollar versus rest of the 401k industry).

Frobenius norm

A quck intro on how to calculate Frobenius norm or 2 norms

1
2
3
4
X = np.array([[3.,5.,8.],[4.,12.,15.]])
norms = np.linalg.norm(X, axis=0)
This results in
[  5.  13.  17.]

This the equation
Frobenius_norm

To calculate this
SQRT ( 3**2 + 5**2 ), SQRT ( 5**2 + 13**2 ), SQRT ( 8**2 + 15**2 )
The output is 5, 13, 17

Business Technology Engagement

In the digital age, for business to achieve competitive advantage, time to market, strategic advantage means business partnering with technologists to develop software solutions that provide the differentiation advantage. Typically this boils down to – either we go find a custom off the shelf solution or build technology solution ground up or enhance existing solution, that fits within the existing architecture, to move bits and bytes around. We do this to create or enhance the business processes to deliver on the differentiation advantage. Putting on a simplistic lens, there are two ways the technology and business team work together to determine the solution – one is what I call (1) transactional engagement and other is (2) involved engagement. Without driving into the details, common sense, suggests that involved engagement is better than the transactional engagement. But often times we find the modus operandi is transactional in nature resulting in sub par outcome with lot of non productive “us/them” discussions.

To draw a parallel, a transactional engagement is similar to a car servicing experience. In general, most of us, average car users, we take our car into the dealership because of a problem. We experience the problem (like not getting enough mileage or not having enough power etc) but we have little idea what is under the hood and what has to happen to address the problem. We trust the service technician whom we regard as an expert. So the expert looks under the hood and come back with a list, this is good, that is bad and total cost of $x. While most of us are not very happy with the $x but then we take it as cost of everyday life. A very similar set of interaction occurs between technology and business. The business, like a car user, defines a problem statement which is loaded with assumptions and misunderstanding because they don’t know what is under the hood. The tech guy, like a mechanic, pops up the hood figures out what needs to be done and then rings up number sometimes running into million $. The business with limited alternatives agrees and embark through the work. The tech guy goes to work, and unlike typical car service transaction, the half-baked understanding and assumptions starts falling apart here and once the work is done, there is an even bigger bill for the business to foot. Often times, the finished solution is not exactly the business had in their mind in the first place. Us, seasoned technology folks, all have seen this play out. This is the classic transactional case, where the engagement is transactional. I have a problem, you have a solution, it cost certain money, i pay and i get the solution. Most of the time the solution is not what the business wanted.

On the other hand, the business and tech can try alternate approach to engagement. To draw a parallel, this approach would be akin to adding a new media room in the house. In the case the person who is looking to put in a new media room takes on a very active interest to make sure everything is built as per desired specs and ends up achieving the desired outcome. This, as it turns out, is a very collaborative process where in the builder/architect is intimately involved with end user often time adapting where it is possible. As the product is shaping up the user can walk around and get an intimate preview of how things will finally shape up. Here the constant engagement and ongoing fine tuning is the key. So any structural mods that needs to be done gets identified and done early in the game to keep the cost under control. The end is practically left for cosmetic changes. The same idea holds true for software development as well. The business needs to closely work with technology, get behind the scenes and help identify changes, especially the fundamental ones, early on in the process. This will involve the business and tech to walk around the system and make sense as to how things are coming together and work together to piece it together. The Agile SCRUM process which is replacing the traditional waterfall in a rapid manner provides opportunity for the business to be closely engaged but then this engagement needs to go even further. Agile SCRUM addresses the feedback aspect with the process. What needs to happen more than Agile is the learning. The business need to get more in depth understanding of technology and process, and technology teams need to get much more comfortable with the business. With learning and process, marked by tight collaboration and frequent feedback, the probability of achieving the desired outcome is for certain high.

To be truly successful in any sort of software implementation and software related change, it is important for both the business and technology to work together in close partnership. The technology team need to speak the language of the business and business need to develop courage to look under the hood. This is what will ultimately ensure that multi million dollar investment bears dividends. Without such active engagement… it will be one troubled deployment after another…

Battle of Trafalgar & Corporate Strategy

On 21st October, 1805, off the coast of Cape Trafalgar, Spain; the Royal Navy commanded by Admiral Horatio Nelson was facing off the combined fleet of French and Spanish navies. Commanding a fleet of 27 ships, Lord Nelson was up against 33 Spanish-French fleet commanded by the French Admiral Pierre-Charles Villeneuve. The French Admiral had a 6 ship advantage which, in those days, practically determined the outcome of naval engagement.

The naval battle formation of that time involved the opposing fleets lining up against each other in a close parallel formation and then striking each other with cannon fire followed by hand to hand combat until one side was clearly defeated. Admiral Villeneuve, following the established norm, organized the Spanish-French fleet into a ragged line formation. However, Lord Nelson deviated from the conventional naval orthodoxy and attacked the combined French-Spanish fleets orthogonally instead of lining up along side. Lord Nelson easily breached through the French defenses and inflicted a devastating defeat to the combined fleet.

The battle of Trafalgar was one of the most decisive battles in the history of Naval warfare. England’s victory in this battle established it as unfettered sea power for the next 100 years. While the strategy and technique used in this battle has been studied by military strategists, the learning from this decisive battle can be equally applied to the corporate strategy as well.

In today’s business environment, the advantages of the well managed firms that have been around for many years can be compared to that of the Franch-Spanish fleets. However, the established giant’s might and dominance are being threatened by newly minted upstarts. These new generation of companies fueled by easy capital, global talent and access to sophisticated and unlimited computing power are going against the traditional orthodoxy that defined the success of the entrenched firms. These companies are introducing innovative new products and services faster, better and sometimes cheaper than the incumbents. Take for example, Uber, a firm that was established only in 2009 and went operational in 2011. It is threatening the traditional taxi service industry which has been operating ever since the first model-T rolled out of production lines over a 100 years ago.

Taxi operations is one service industry where status quo reigns supreme. Investments in technology made by established taxi services have been marginal resulting only in incremental changes. Rigid pricing structure and poor customer experience and satisfaction has become the norm. The competitiveness was limited to non customer value creating aspects like who gets to drive a cab etc. In NYC, for example,medallions that gave the driving rights wereauctioned for millions. The inward looking industry did not focus on things that mattered like better customer service, better use of technology or adopting flexible pricing strategies. Uber changed all that! Founded with a vision of providing great customer experience, the company leveraged mobile technology to deliver exceptional service, convenience and information. Uber also innovated by adopting a market driven pricing strategy based on demand and supply. The entrenched taxi service companies while having resources were stifled by conventional orthodoxy in spite of being aware of technological changes happening all around them. They were unwilling to make the change or move fast enough to adapt to the new reality.

In conclusion it is very important for business leaders not to get caught up in the web of conventional orthodoxy because the Lord Nelsons of today are far more ruthless, fast and agile. These nimble competitors not bound by the traditional rules and are bringing newer ideas to the market faster, better and cheaper. The leaders of the established firms must not be afraid to pull a “nelson” on their own business or product lines in order to survive and thrive. They need to embracethe idea of cannibalizing their product lines or to turn the existing business model upside down if necessary. If they are unwilling or unable to do it, the nimbler competitors unconstrained by conventions will come along and destroy the firm transforming the business model inside out. While the real battle of Trafalgar ended with emphatic victory for England, in this new flattened global stage the battles continues ….

Innovation

lj6wqtlj

 

These days, hardly a day goes by without hearing the word innovation. We hear how innovative companies disrupts well-heeled incumbents and how some well established, large companies have transformed to develop an innovative edge in the hyper-competitive market. The innovation reality, however, is limited to startups and select few larger companies. For the majority of larger and older companies, the ground realities is actually “business as usual”. Some of them tend to innovate by acquiring others who have innovative edge rather than transforming their own core operations. The employees, in these companies, come in to work, stick to what is expected of them while aligning themselves as much as possible to company’s vision, mission and goals. These employees tend to operate in a centralized hierarchies with command and control structures. Many of these employees still endure the oppressing bell curve rating during the year end that determines their compensation and year end bonuses. In other words these employees are stuck in the box, constrained, unable to contribute to the innovation frontier.

However, there is an ever increasing desire among corporate leaders and organizational psychologist to figure out theory & practices to get the people off the “business as usual” trance and enable the ordinary to do extraordinary work. Various academic studies has firmly established the correlation between employee engagement and motivation to producing creative output. As a mean to further the innovation agenda, companies look to hire intrinsically motivated, engaged and talented individuals to help transform the organization. They do this, because it is clearly established that intrinsically motivated employees are driven and are more conducive to creativity.

Intrinsically motivated employees are highly engaged and produce creative output only when they operate in an environment that involves task variety, decision rights, and intellectual freedom. But in reality, companies that hires these intrinsically motivated top talent, end up making them work in an environment that has remained the same for many years. In the environment process & procedures reigns supreme, decision rights are restricted or at the best limited and organizational red tapes are a norm that constantly reminds what should not be done. In this environment, the employees finds the work banal, constrained and unimaginative. In such a settings the intrinsically motivated employees have two choices (1) Confirm to become a “clocker” or (2) leave to another company where s/he can thrive. These companies, that is seeking the best from the employees, tends to motivate them through extrinsic means which involves year end appraisals and a bell-curve rating which provides support for annual “rank and yank” round-tables. While from a company standpoint the “reward system” seems fair and logical it, this form of motivation leads up to detachment and overall reduction in creativity.

So the natural question is how entrepreneurial companies are able to leverage the same pool of talent available to larger ones but still execute on transformational idea better than larger ones? The answer is the environment in which people operate. In an entrepreneurial company, the leaders are not risk averse. They encourage risk with fail-fast, fail-often and learn fast approach. This type of environment is unthinkable in a command & control hierarchy where employees responsible for any failure are often viewed unfavorably relative to others who are risk averse. The year end rank and yank process ensures these “bad apples” are yanked out. The second attribute of an entrepreneurial firm is low barrier to communication/collaboration. Collaboration happens in a highly networked context where information and ideas flows freely and are fully vetted out. In case of larger firms, collaboration is replaced by meetings where the problem shifts from solving problem at hand to finding time that works for all the participants to meet. The third attribute that makes small entrepreneurial firm effective is speed of decision making. In a small flat hierarchy it is easy to talk to the decision makers and get to decision faster. Speedy decision enables speedy execution. In large organizations, however, decision making takes months and sometimes years because it needs to be vetted out every manager in the hierarchy effectively creating bottleneck as decisioning process moves up and down the chain. Employees also spend inordinate number of hours preparing justification powerpoints that are tailored to whims, fancies and format preference of executives. The fourth attribute is task variety. In smaller firm, an employee perform multiple roles and has the freedom to try out new things without fear. He/she has unconstrained, multi-dimensional learning opportunities. In larger firms, however, employees are generally stuck in a straight-jacket with very little elbow room to try anything new. Big firm risk aversion culture means, the firm would rather hire a new employee from the street rather than empower an internal employee with desire, ambition and ability. The fifth attribute is nimbleness. The small firm is able to react fast to the market condition and incorporate market feedback rapidly. The distance between market feedback provider and the person who is able to incorporate the feedback is small, and hence the probability of information dissipation due to “chinese whisper” is low. In case of large firm, market feedback flows through large hierarchy chain and takes many days or even months before it gets to the person in charge of implementing the change. In addition, there is high probability of feedback gets mutated along the way.

Even in the case nimble entrepreneurial firm, as the innovative product takes a foothold in the market, the firms starts focusing on scale economies. With scale the firm gets larger, the founders start to lose “control” and the work environment mutates. The environment or culture that reflected the founder entrepreneurial spirit now starts to become more “organized” and banal. The differences in resources, collaboration and coordination costs slowly increases. Leaders, to achieve better control, tend to put in place process & procedures which results in increased bureaucracy and routinization of work. This standardization produces a degree of “command & control” generally results in diluting individuals’ sense of ownership and responsibility for their work. The routinization, however, enables firm to transfer work to cheaper location further diluting individuals sense of ownership. In such set up, the company starts to loses the innovation advantages that was primarily because of the intrinsically motivated and highly engaged employee. With employees now operating in a production line mode, innovation becomes a jargon and a lofty goal.

In the age where innovation edge dictates if the firms will survive the next decade, larger companies need to start doing things differently. These companies need to make best use of their assets which are not buildings, not products, not equipments but their people. To control cost and to be fundamentally be innovative, companies need to fundamentally rethink how the organization is designed, how work is conducted, how engaged are employees, and how people are rewarded. Without such inside-out people transformation, the only other way to innovate is by acquiring small innovative trendsetter. But the best long term strategy would be a inside out metamorphosis rather than externally induced transformation. In my next article I will provides some of my ideas on how big firms could re-do their organizational architecture to better compete.