Snapshot | A decade of AWS and re:Invent
Ten years ago, AWS started with a small group of people in an office in downtown Seattle. The AWS services offered at the time were basic by todays standards. But they helped Amazon and other early customers focus on building their business rather than managing hardware and software infrastructure. AWS has become the leading global cloud computing provider, powering hundreds of thousands of customer websites, mobile apps, and IoT devices. As we approach the 10th anniversary of AWS re:Invent, I reflect on how AWS has evolved over the last decade and what that means for financial institutions going forward.
Early years: A pitch to CFO’s and their IT departments
The financial crisis was a boon for AWS. CFOs recognized the opportunity to save money, shift capex from expenditure towards opex and avoid getting locked into long-term capital depreciation schedules, over-provisioning or constrictive leases that would cut down on their flexibility in the future.
Amazon held the very first AWS re:Invent summit in 2012. AWS first began by hosting their re:Invent conference as a global event for AWS users and AWS technology partners to come together. It also allowed those initial 6000 odd primarily technical attendees interested in using cloud computing to learn about AWS firsthand from experts within the industry. The secondary emphasis was providing demos of its capabilities directly to customers who were unsure about using cloud computing for sensitive data.
In 2012, AWS had a minimal presence within financial services firms. S3 and EC2 were the primary services offered — mainly offering virtual machines with blob storage. Financial services firms spent a significant portion of their technology investments on data centers. Many with sophisticated resiliency capabilities to meet regulatory and institutional customer expectations. Large corporations had limited attraction to the basic offering with little differentiation against their own powerful data centers.
This nascent capability then evolved to include clustering and the EC3 load balancer, the beginning stages of offering complex load balancing capabilities out of the box without the need to pay or configure Cisco and IBM load balancers.
Despite the intrigue of the new ease of maintenance, many financial services firms were wary of the Cloud. High availability, security and privacy concerns were top of mind. AWS knew that it would have to provide a transparent and easy-to-use service to gain the trust of financial institutions. AWS designed specific services such as AWS Shield, AWS WAF, and AWS Glacier, significantly influenced by feedback from financial services firms. These cloud security tools allowed AWS customers to feel secure using AWS.
2016: The midpoint — regulators come on board
By the 2016 AWS re:Invent, cloud growth was coming from startups and large enterprises looking to keep up with the latest technical innovations on the Cloud.
By 2016 financial services regulators had also begun to embrace cloud computing. FINRA presented a keynote at AWS re:Invent in 2016 describing the innovations that enabled superior regulatory monitoring.
AWS was able to provide what regulators describe as a “regulatory compliant cloud.” AWS compliance with financial services regulations is now recognized globally by many regulatory bodies, including FINRA, the UK Financial Conduct Authority (FCA), and Japan’s FSA, among others.
Robert Palatnick, the chief technology architect at DTCC, once described how the centralized clearinghouse that processes 100 million transactions per day transformed trade processing and analytics using AWS. DTCC is all in on AWS, running more than 20 workloads in a regulated environment that demands resilience, secure storage, and industry-wide collaboration.
Major data center providers like Verizon and ATT began integrating their offerings with cloud-based services to provide an all-in-one solution for businesses. This reduction of the line between traditional data centers and clouds will allow organizations more flexibility to store data and make remote access more effortless than ever before.
AWS GovCloud launched in the US. An AWS region designed to allow U.S. government agencies at the federal, state and local level, along with contractors, educational institutions and other U.S. customers to run sensitive workloads in the cloud.
The Front Office in the banking services, micro-services of Sales and Servicing, CRM and the Channels, customer-facing wealth management applications took around 32% of the Cloud adoption in deals assessed by Gartner from 2014 to 2018
Core banking systems, Payment processing, Back-Office collaboration, and AM and VW systems represented 31% of the cloud migration from 2014 to 2018
2020: The end of the first decade — powerful compounding advantages take hold across the sell-side, buy-side and corporate banks
By 2020 almost all of the global financial institutions were using AWS.
The conference attendees count at re:Invent 2019 grew tenfold from the very first one in 2012. Nearly 60,000 attendees traveled to Las Vegas to hear Andy Jassey and Werner Vogles speak. I was one of them. And in that period, for financial services, particularly, AWS unleashed a wave of innovations and products within financial services.
The scale and size of AWS allow financial services firms to innovate and deploy at scales inconceivable in their own data centers. AWS now has the largest compute footprint in financial services, larger than any other cloud provider. AWS can provide a “regulatory compliant cloud,” making it attractive for use by financial institutions around the globe as they can do business with all of their customers from one location regardless of where those users are located.
When Goldman Sachs rolled out their Transaction Banking offering on AWS, they took very extreme scaling measures where every corporate account workload was essentially running in its own container. The infrastructure to replicate running 1000’s of containers in their own data centers would have been inconceivable a decade ago.
At this year’s conference, Goldman will be speaking about reinventing financial data analytics. The bank rearchitected its proprietary data and analytics capabilities using AWS services to make it easier than ever before. They were able to combine different types of information in minutes rather than hours or days, as was previously required for complex tasks like this one.
JP Morgan Chace had already migrated many trade desks to AWS and achieved improved performance.
The bank’s Athena platform provides technologists, quantitative analysts, and risk managers with a consistent, cross-asset portfolio of models, frameworks, and tools to build financial applications. JPMC decided to accelerate Athena’s journey to the Cloud. The Athena team adopted cloud-native services, including EKS, S3, RDS, and ElastiCache for Redis, to enable customers to evaluate risk across a broader set of market conditions and scenarios.
Vanguard chooses AWS to help modernize its traditional, heavily virtualized tech stack, big data platforms, monolithic applications, and a PaaS running microservices
This year Vanguard will be speaking about personalization at scale with Sage Maker. Vanguard is using machine learning (ML) on AWS to solve one of the most pressing challenges in financial advice: personalization. Vanguard’s ML engine uses AWS services such as Amazon SageMaker to optimize rule-of-thumb recommendations with more sophisticated financial advice to maximize retirement wealth and spending for investors. Vanguard will share how training the ML models on large SageMaker instances, including more than 120 instances for data generation, made the unsolvable solvable, accelerating project completion time to less than one month and lowering the cost from $5 million to less than $200 thousand.
Wellington Management executed a multi-year strategy to exit its physical data centers by migrating commercial and custom applications. The firm completed the migration of all physical data centers by 2019 utilizing Virtual Private Cloud, Direct Connect, EC2, ECS, Lambda, Redshift, Relational Database Service, and others.
Technical innovation unlocks possibilities for product managers
EC2, RDS, S3, Cloudfront, VPC, SNS, Beanstalk, Lambda, Autoscaling, and IAC lead the pack in terms of most used AWS services.
However, the rate of innovation within amazon web services unlocks new opportunities for product managers that may have been previously inaccessible.
Here are some financial services use cases that benefit from key AWS services:
- Amazon Timestream — Enables high volume, high-performance use cases, intraday pricing, technical analysis of price levels
- Amazon Textract — Enhances KYC use cases such as extracting data from customer ID cards
- Amazon QLDB — Provides cost-effective distributed ledgers which are cryptographically secure and supports capabilities such as audit trails and intercompany trades
- Amazon Forecast — Leverages time-series data to discover the hidden patterns in user data to predict customer behavior, new account sign-ups, expected withdrawals, and deposits
- Sagemaker RL — Improved reinforcement learning capabilities used to train models to maximize factors relative to a customer’s trading strategy
- S3 Object Lock — Simplifies compliance toward regulatory rules enforced by the SEC, FINRA, and CFTC with the ability to write items into storage such that records cannot change them
- Dynamo DB — Introduces ACID-compliant transactions that are critical for financial applications
- Amazon Personalize — Increases customer engagement by enabling tailoring recommended financial products to target customers by leveraging amazon.com’s recommendation engine
- Amazon Managed Streaming for Kafka — Enables real-time risk and pricing use cases by leveraging streaming trade and market data
Growth comes with risks
The astounding pace of innovation does come with its risks. Since starting with less than 30 services a decade ago, AWS is expected to announce a library of nearly 300 services at Re-Invent in 2021. Some of this growth represents new capabilities that allows product managers to offer new services to customers. Others however change previous paradigms that require re-thinking. The net is certainly a new unique challenge for product managers and their engineering counter parts.
The breadth of services can be hard to keep up with
Today it is far less likely that any one person can know all of AWS. The depth and breadth of services are nearly encyclopedic in their scale. This scope can present some challenges to product managers. For example, accurately predicting underlying costs of services use.
Lightsail is meant to cover up some of the getting-started complexity of provisioning new EC2 machines. Elastic Beanstalk is another AWS service that came along to give developers a shortcut for making deployment environments. But even with AWS making these helper services, there’s still a need to understand what’s going on. For example, when you look at your bill, and you’ve been using Elastic Beanstalk, your statement won’t have a single line item for your Beanstalk usage. It will have lots of line items for all the core services that Beanstalk itself is composed of. If you don’t have at least some inkling of what those are, then you won’t understand your bill, and you won’t be able to predict it.
Developers need to be versed in infrastructure and security
The collapsing of silos within the technical department introduces an even greater need for developers to be even more mindful of security vulnerabilities and resiliency in their application designs.
Managing cloud resiliency to ensure applications can withstand outages and fail gracefully. Allocating time to training and upskilling team members. Rushed experimentation can have grave consequences — such as leaving S3 buckets open for anyone to access, the OS upgrade issues faced by Capital One amongst many other data hacks on cloud.
Compliance approaches need to be rethought
In the financial sector, some compliance requirements are prescriptive, “For example, the Payment Card Industry Data Security Standard (PCI DSS) requirement 8.2.4 obliges companies that process, store, or transmit credit card information to “change user passwords/passphrases at least once every 90 days.”
In contrast, compliance requirements for managing operational risk can be subjective. Regulators take a principles-based approach to setting regulatory requirements. Guidelines that FI’s are expected to consider when they design and deploy products.
A simple example “My data centers need to be X kilometers apart” intends to minimize location-based disruptions such as natural disasters. In the Cloud, vast physical distance separation is an anti-pattern. AWS designs global infrastructure to balance the physical distance between AZ’s within an AWS region to achieve high availability but the right proximity to support synchronous replication
Cloud service provider lock-in fears grow
As AWS and many of its competitors move further upstream, lock-in risk dominates the conversation between CTO’s and their risk teams. Many S&P 500 CTOs see the importance of cloud independence as an architectural and strategic mandate. The strategy that is continually evaluated is how to develop multi-cloud to avoid getting locked in while ensuring that developers are productive and taking advantage of new innovations. They are essentially seeking to separate the underlying infrastructure from software development.
Looking ahead to AWS re:Invent 2021 and the beyond
The future looks bright for AWS. AWS is a company that has evolved significantly over the last decade to help power innovative financial services products.
The Cloud’s evolution is a huge part of what makes it such an interesting system. The Cloud has evolved into something more than just storage space. The vast computing power enables hyper-distributed intelligence where any business can be built or transformed by this ever-changing internet resource capable of storing data and processing transactions at lightning speed to its connections all over town.
“The greatest shortcoming of the human race is our inability to understand the exponential function.”
― Albert A. Bartlett
Our human cognitive biases block us from comprehending the power of exponential growth. Albert Einstein too once observed that compounding interest was humanity’s greatest invention. Only in hindsight is its impact clear. A decade ago, the breadth and impact of the cloud transformation journey led by AWS couldn’t be comprehended.
As we look ahead, we can only wonder, what this next wave of exponential growth will bring.
Sources:
- Amazon Web Services. “AWS Well Architected Framework for Financial Services.” AWS Well Architected Framework for Financial Services, 2020, .
- “Cloud Heat Map for Banking and Investment Services, 2019.” Cloud Heat Map for Banking and Investment Services, 2019, gartner.com.
- “Disaster Recovery Compliance in the Cloud, Part 1: Common Misconceptions.” Amazon Web Services, 14 Sept. 2021, .
- Lowe, Ned. “Re:Invent 2018 — A Financial Services Take — Ned Lowe.” Medium, 3 Dec. 2018, .
- Vellante, David. “Breaking Analysis: Cloud 2030…From IT, to Business Transformation.” Wikibon Research, 28 Nov. 2020,
