# Wells Fargo Foundation Model: Application to Fraud Detection at Scale — Freddy Lecue — session 2026-10-13T14:40:00.000Z → 2026-10-13T14:58:00.000Z _53 transcript lines · 36 slides_ ## Transcript Alright. Thank you, guys. My name is Felipe Pew. I'm part of the web partner team. I'm leading the PringQ AI organization there. And I'm going to show you the work we've been doing lastly on building, our first foundation model, Wells Fargo foundation model. So I'll explain why we did that, how we did it, and what we accomplished in doing so. So so first, Foundation model, it doesn't necessarily need to be language based. I mean, we're all playing with, CLODE and ChartGPT and others. They're all language based foundation models. In financial services, we also have language, of course, from a structure... Unstructured representation data. But we also have a lot of structured representation. A lot of these data is tabular data. We have a lot of information from our customers. We have information in investment banking, consumer banking. And, all of these data is data that we have in in Wells Fargo, and all big financial organizations have too. And, we want to take benefit of all of these huge amount of data that we have. And it's not like you just take it and put it in a in a language model and get the best of it. Of course, you can get a lot of benefit out of it, but it's not necessarily the best way to do it. So so what we decided to do is to look at the data we have, all the transactions. All the customer transactions that they're doing with their debit card, credit card, digital check, all of those coming from many different channels and see if we can build a foundation model that we can use and fine tune for downstream tasks like credit scoring or that you can use it for fraud detection, anti money laundering, do marketing. So how we can build one backbone that you can use and streamline across downstream tasks? So now you're moving from many models We siloed across different organizations in really one backbone, and every organization can just reuse because you learn the representations of customers' transactions at at scale. So that's the intent. And just follow-up on it. You may say, Hey, that's tabular data. Why do you just don't use XGBoost and the usual suspect? We did that. This is the model we had in production. And... But the model I'm going to show you is something that get... That beats XGBoost, even from a latency perspective. But also from a, from a fraud detections, we get much better false, reduction of false positive and better detection rates. So it was a very good lesson learned in terms of having better model using foundation model. And foundation model mean transformer blocks. And that... That's what we build. So, so we have, as I mentioned, billion of transactions, on a monthly basis. Our customers interacting with the Credit card, and they do check, and they do different sort of transactions. And the the question is really how we can use this information at at scale and not for one and every single downstream task. So I'm going to cover a bit of the data, the representations, the architecture that we use, how... That... What the result we get, and the lesson we learned in building this. Alright. So left hand side, language foundation model, the processing Language tokens, and then the the goal is to learn the representation that's going to predict the next token. And then you have, you have supervised fine tuning, and then you do LHF, LRVR, and all of the great post training process to get better at the task that you want to. Right? And that's for languages. That's great. But what if your data is very much more specialized in terms of representation? So that's where we build this transaction foundation. Model. So instead of having languages as the base, the base is now transactions, transaction from our customers. And, you still try to predict the next thing. And here, the next thing is the next transaction or it's the next token in the transaction using all of the channel that you have. But that's not the end goal. I mean, the similarly to language model where you try to predict the next token, that's not the end of the story. Right? You learn the representation. And then you use that representation And then you you you find And you post train it for any other task. Right? So it could be classification task for languages. But in our context, for transaction foundation model, it could be used for, credit line increase, could be used for doing fraud detections and flagging bad actors from an anti money laundering perspective. And the goal is to learn this, relationship between transactions over time and transactions support customers. That's really the intent. So the intent number one is you do pretrain from scratch. I'm saying it again because people sometimes confuse. We pretrain transformer blocks, at scale from scratch using our, transaction data. So the way... The zero rate that we use from open weights, we just use the architecture, and then we we pre train it from scratch using our GPU cluster. And then we train for a couple of days and and weeks, and and then you do the the downstream task of, of post training. But the goal here... And you pre train it. For predicting the next token or the next transactions. And you you try to to do this, right, you try to predict the next the next pattern pattern of the customer, but that's not the intent. The intent is to do the downstream task later on. And in terms of scale, I mean, we we operate at a large scale. So we have billions of transactions, a million of accounts, hundreds of transactions per client on a monthly basis interacting with their debit card, credit card, or doing check. And the the five main dimensions we look at is really who is doing... I mean, who is processing and who is asking on who is providing a service. So will it the customer or the merchant? And the the what. So, I mean, what what do we what do we buy, what do we transact, And how? So we channel. Is it credit? Is it debit? Is it digital? When is it happening? And where is it happening? And all of those, they're basically breaking down in Android of features. And, if you look at the existing model we have in production, there's a lot of... I mean, people that build those model in similar organizations may, may understand that they build a lot of very hard crafted, uncrafted engineering, future engineering efforts on that. And that's required a lot and a lot of time. So we step back and look at the way the raw features we need, not the one that been prebuilt and postbuilt for doing the downstream tasks, but try To understand which raw features. And we ended up with basically Android features, even if, at the end of the day, we had thousands of them. But we tried to find which ones are the raw information you need. And then we let the architecture or the transformer blocks to figure out the relationship between those core features. And, as you can see, we do have a bunch of features on those five dimensions. And the vocabulary is very unique. To the financial services set up. So... And that's the reason why, I mean, we didn't really use, existing pre trained model, because those... The vocabulary that we use is very, very specific to the work that you want to do. And even if you merge, while all digital and checking and, saving and, all of those sort of transactions, even if you merge all of those, you still end up with a very limited number of features and character. So at that scale, then the question is, well, how how you work with that? So so we started... Everything is tabular data. I mean, there's no secret it's tabular data. And then the goal is to understand how you're going to capture the vocabulary because you do have numerical values, you have categorical values, you have, cat... I mean, you have information like zip code. Do you encode each and every zip code in a token? Does it make sense? So there's a lot of effort you need to push in understanding. The vocabulary that you need and the vocabulary that you want also to scale. Because at the training, if you have a too huge vocabulary, it would be kind of messy, to train such such a model with a vocabulary which is far, far too big. So so there's a lot of effort in understanding this, value to tokens. So which information do you encode in a token? And then you have the token to add, the tokenizations, and then the embedding lookup. So then when we have a transaction coming up, you do have the vocabulary. So it's a, basically, it's a lookup. And aligning her transactions and all the characteristics of that transactions to to a token and an ID and then an embedding. So just to give you a number... Well, some numbers. So we have, yeah, quantitative, qualitative, categorical, of different kind. Right? So categories with only eight types, some with much bigger types. You have transaction amount, which is a big deal. You have the delta time between transaction. So you have anything related to IP. Postal code. So just to give you example of, of variety and heterogeneity of each and every characteristic could be encoded in a binary token or could be encoded in a much larger number of tokens. And Based on distributions, for instance. So, architecture... So so here just, what we... So we use the the chinchilla, well, mathematics behind the... To define how we build it. Right? I mean, initially, because we we had no clue how it's going to work in the tabular space, even though there are existing, well, open weights and open solutions for doing, tabular data foundation models. So we we we start From what they learn and adapt it to what we needed to do. And that gives you an example of embedding dimensions, number of decoder blocks we used, and, all of the the recipe from a... From an architecture perspective. So we have... And as you can see, we're not trying to build a trillion parameters model. We don't try to compete with any of the, the the leaders in the language models because we're working really and focusing on transactions. Right? Transaction Foundation models at billion level. And one one billion is because that's the model that give us, that was the... Well, kind of the smallest and give us the highest boost. And now we're working on a... On the next versions at 11,000,000,000. But that's good enough to to give a lot of, error rate, from a false positive reductions for downstream tasks like for detections and detection... Unit detection rate. So... And the and the the type of architect To the decoder only architecture, that's basically for predicting the next the next token or the next transactions. And that's really the the pretraining. So, so here, we initially used 20 times the number of parameters to define the number of tokens we might need. We ended up actually 10 times factor factor than that. So by doing a few experimentations, we needed much more, much more tokens per parameters. I mean, this is this is kind of the initial experimentation we did. So I'll pass on Because this is the the usual suspect from a, objective functions of causal language, modeling and math language modeling type of optimization work. And and, yeah, from an architecture perspective, nothing really... I mean, fancy here. The the the... Again, the most complex part was really the tokenization and then and then that update and doing the pretraining and then how you do the fine tuning. So if... I would say the the recipe or the secret sauce is really how you do Tokenization and how are you going to do the post the post training. But anything from a training perspective, it's, it it it... The question is just how many how many transaction you're going to put in your in your context or in your history for for doing the training And evaluation, perplexity based, accuracy accuracy dashboard on downstream tasks, that's that's the usual aspect. And, so... Yeah. So use mixed precision, distributed data parallel. Maximize macro batch size, get faster convergence, yeah, using M1, Rope, usual, and memo efficiency, flash attention, activation checkpointing. I mean, here, there's nothing really new. It's just, we use the the the the advices from the from the best and from the the one that built even much bigger models. So, I'll pass on this one. Time is short, but, I wanted to spend some time I'm, yeah, on the results. So, yeah. So the... When you build this foundation, that foundation is really... Is is off... Is really... Is is on your backbone. It's something you can use for downstream task. And, the downstream task was to do fraud classification. So you have a transaction and then you try to predict if that transaction is going to be fraudulent. So in the model we have in production, it's an XGBoost model, and it's trained On large amount of transactions, which is, fraudulent and non fraudulent. And the missing piece is that was the, relationship between whatever transaction you try to predict, try to classify, and the relationship with this transaction with all of the past transactions. And that information is kind of gone when you use an XGBoost model, unless you encode it in a very hard way, which is which is not so easy. But if you use transformer architectures, you do have this nice Attention from, from transactions to past transactions. And that's basically what help, this is what we think. This is what will help, pushing the fraud detection rate much higher. But at the end of the day, the the downstream task was to say, here's the sequence of transactions. I have someone just using the debit card. ## Slides ### 00:00:22 # AI Engineer Europe 2027 ## Event Details - **Location**: ExCeL London, UK - **Dates**: February 17-19, 2027 (3 days) - **Description**: Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. - **Website**: [ai.engineer/europe](ai.engineer/europe) [Black and white photos of London landmarks including a classical building, the London Eye, and Big Ben with the Houses of Parliament] ### 00:00:50 # PRESENTED BY ## arize View Full Schedule at ai.engineer/nyc/2026/schedule [Logo for Arize, featuring a pink upward-pointing arrow icon] ### 00:01:28 # AI Engineer Europe 2027 ## ExCeL London ### 02.17 - 02.19, 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. [ai.engineer/europe](ai.engineer/europe) [Black and white photo of a grand building with an ornate facade] [Black and white photo of the London Eye over the River Thames] [Black and white photo of Big Ben and Westminster Bridge] ### 00:01:48 ## SILVER SPONSORS - Anchor - apify - AXD CAPITAL - Chainguard - CLOUDFLARE - DE Shaw & Co - Expo - extend - FLOX - Gravitee - Judgment Labs - Kimchi - MINIMAX - NIMBLE - Pydantic - RELAI - SENTRY - specific ai - supabase - SurrealDB - tetrate - UNION View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Logos of the listed sponsor companies] ### 00:02:16 # The **Highest-Signal** Technical AI Conferences AI Engineer hosts AI developer conferences around the world for engineers, breakout startup founders, and AI architects advancing AI engineering at the world's leading technology companies. ## Find Out What's Next Scan the QR code or visit [https://ai.engineer](https://ai.engineer) to subscribe to our newsletter and stay up to date with the latest news and events. [QR code linking to https://ai.engineer] ### 00:02:48 # AI Engineer ## NEW YORK # WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Illustration of the New York City skyline at night, featuring prominent buildings like One World Trade Center, the Empire State Building, and the Statue of Liberty] ### 00:03:15 # AI Engineer ## NEW YORK # WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [New York City skyline at night with prominent skyscrapers and the Statue of Liberty] ### 00:03:51 # Wells Fargo Foundation Model: Application to Fraud Detection at Scale Lessons from building a large-scale transaction foundation model on **billions of banking transactions**. ### Freddy Lecue Head of Frontier AI Model, Managing Director Wells Fargo 1. **RAW DATA** billions of bank transactions 2. **FOUNDATION MODEL** one general-purpose transaction backbone 3. **SPECIALIZED MODELS** fine-tuned for fraud, Anti-Money Laundering, decisioning, ... 4. **BUSINESS IMPACT** lower fraud loss, less analyst toil [Horizontal flow diagram showing four steps: Raw Data, Foundation Model, Specialized Models, and Business Impact] ### 00:04:20 # FOUNDATION MODELS What they are? Why they matter? What they unlock in finance? ### WHAT IS A FOUNDATION MODEL? A single, **very large model** pre-trained on huge amounts of **unlabelled data**, that can then be **adapted to many downstream tasks** with relatively little task-specific data. - PRE-TRAIN: self-supervised on raw data - ADAPT: fine-tune or prompt per task - DEPLOY: many products, one backbone ### THE PARADIGM SHIFT - BEFORE: one task, one model - NOW: one backbone, many tasks (Foundation Model) ### WHERE FOUNDATION MODELS LAND IN FINANCIAL SERVICES — any task framed as a sequence-of-events, document, or natural-language exchange - **Fraud & AML** score every authorization from raw transaction history - **Credit & decisioning** richer risk signals from full account behaviour - **Customer service** agents that read accounts, policies and complaints - **Markets & research** summarise filings, analyse earnings transcripts - **Operations** reconciliation, KYC review ONE BACKBONE, MANY DOWNSTREAM TASKS — in finance, the most data-rich signal is the **transaction stream**. ## Wells Fargo Foundation Model: Application to Fraud Detection at Scale Freddy Lecue / Head of Frontier AI Models, Managing Director WELLS FARGO [Diagram showing a three-step process: PRE-TRAIN, ADAPT, DEPLOY] [Diagram illustrating a paradigm shift from multiple individual models for tasks to a single Foundation Model serving multiple tasks] [Five icons representing different financial services applications: Fraud & AML, Credit & decisioning, Customer service, Markets & research, Operations] ### 00:04:51 ### FOUNDATION MODELS ## What they are? Why they matter? What they unlock in finance? ### WHAT IS A FOUNDATION MODEL? A single, **very large model** pre-trained on huge amounts of **unlabelled data**, that can then be **adapted to many downstream tasks** with relatively little task-specific data. - **PRE-TRAIN** - self-supervised on raw data - **ADAPT** - fine-tune or prompt per task - **DEPLOY** - many products, one backbone ### THE PARADIGM SHIFT - BEFORE — one task, one model - NOW — one backbone, many tasks ### WHERE FOUNDATION MODELS LAND IN FINANCIAL SERVICES — any task framed as a sequence of-events, document, or natural-language exchange - **Fraud & AML** - score every authorization from raw transaction history - **Credit & decisioning** - richer risk signals from full account behaviour - **Customer service** - agents that read accounts, policies and complaints - **Markets & ### 00:05:19 # FOUNDATION MODELS ## What they are? Why they matter? What they unlock in finance? ### WHAT IS A FOUNDATION MODEL? A **single, very large model** pre ### 00:05:56 # FOUNDATION MODELS ## What they are? Why they matter? What they unlock in finance? ### WHAT IS A FOUNDATION MODEL? A single, **very large model** pre- ### 00:06:18 # FOUNDATION MODELS ## What they are? Why they matter? What they unlock in finance? ### WHAT IS A FOUNDATION MODEL? A single, **very large model** pre ### 00:06:55 # From raw transactions to production decisions ## Agenda - **01 Data & Representation** - Transactions as 'sentences'. Tokenization strategies for heterogeneous, sparse, sequential records. - ### 00:07:19 # THE MENTAL MODEL A transaction is a sentence. An account history is a document. ## Language Foundation Model > "Four score and seven years ago our fathers brought forth..." Predict the next word - **PREDICTS**: the next word (or sentence) in a stream of text. - **LEARNS**: complex patterns in language (grammar, semantics, world facts). - **POWERS**: summarization, classification, complaint triage, chat. ## Transaction Foundation Model `[BOS] [Wknd] [Δt_10h] [AMT_1] [Purch] ... [EOS]` `[BOS] [Δt_3d] [AMT_2] [Purch] ... [ ### 00:07:55 ## THE MENTAL MODEL ### A transaction is a sentence. An account history is a document. ### Language Foundation Model > "Four score and seven years ago our fathers brought forth..." Predict ### 00:08:21 THE MENTAL MODEL # A transaction is a sentence. An account history is a document. ## Language Foundation Model > "Four score and seven years ago > our fathers brought forth..." Predict the next word - **PREDICTS** the next word (or sentence) in a stream of text. - **LEARNS** complex patterns in language (grammar, semantics, world facts). - **POWERS** summarization, classification, complaint triage, chat. ## Transaction Foundation Model > [BOS] [Wknd] [Δt_10h] [AMT_1] [Purch] ... [EOS] > [BOS] [Δt_3d] [AMT_2] [Purch] ... [EOS] Predict the next transaction (or field) - **PREDICTS** the next transaction — or the next field inside a transaction. - **LEARNS** complex patterns in spending behaviour (timing, amount, merchant). - **POWERS** fraud detection, decisioning, behavioural segmentation, anomaly detection. SAME RECIPE — vocabulary, sequence model, self-supervised objective — just a different alphabet. ### Wells Fargo Foundation Model: Application to Fraud Detection ### 00:08:56 ### THE MENTAL MODEL # A transaction is a sentence. An account history is a document. ## Language Foundation Model > "Four score and seven years ago our fathers brought forth..." Predict ### 00:09:22 # WHAT IS A 'TRANSACTION'? ## Heterogeneous, sparse, sequential — and very, very large - **≈ 10^10** transactions (many months ### 00:09:50 # WHAT IS A 'TRANSACTION'? ## Heterogeneous, sparse, sequential — and very, very large - ≈ **10^10** transactions (many months - one transaction ### 00:10:22 # WHAT IS A 'TRANSACTION'? ## Heterogeneous, sparse, sequential — and very, very large - ≈ 10^10 transactions - many months - one transaction ### 00:10:51 ### WHAT IS A 'TRANSACTION'? # Heterogeneous, sparse, sequential — and very, very large - ≈ 10^10 transactions (many months - one transaction type) ### 00:11:22 # WHAT IS A 'TRANSACTION'? ## Heterogeneous, sparse, sequential — and very, very large - **≈ 10^10** transactions - many ### 00:11:50 # FROM ROWS TO TOKENS ## A five-step pipeline turns tabular records into model input - **01 Tabular records** - raw transaction - rows ### 00:12:22 # FROM ROWS TO TOKENS ## A five-step pipeline turns tabular records into model input - **01 Tabular records** - raw transaction rows (hundreds fields) ### 00:12:50 # VOCABULARY DESIGN ## 97 raw features -> 5,164 tokens across 8 feature types - **Quantitative** - **17 ### 00:13:18 # VOCABULARY DESIGN ## 97 raw features -> 5,164 tokens across 8 feature types ### Feature Types and Token Counts **Quantitative** 1 ### 00:13:52 # 1B-parameter decoder-only transformer, sized by Chinchilla ## ARCHITECTURE — 1B-PARAMETER TRANSACTION FOUNDATION MODEL - **Input sequence**: 1, ### 00:14:19 # 1B-parameter decoder-only transformer, sized by Chinchilla ## ARCHITECTURE — 1B-PARAMETER TRANSACTION FOUNDATION MODEL - Input sequence: `1,024 / 2,048 tokens (account history)` - Embedding dimension: `2,048` - Transformer stack: `16 decoder blocks` - Attention: `32 heads • head dim 64 • masked MHA (Multi-Head Attention)` - Per block: `MLP (Multi-Layer Perceptron) dim 8,192 • residual • layer-norm` - Position: `RoPE (Rotary Position Embedding)` - Objective: `Next-token prediction (Causal + Masked Language Modelling auxiliary)` ## SIZING MATH (CHINCHILLA-OPTIMAL) Optimal tokens ≈ `20 × parameters` - DATA TARGET: `≥ 20B tokens` - TRAIN COMPUTE: `6 × 1B × 20B = 1.3 × 10^23 FLOPs` - WALL CLOCK: `8 × NVIDIA H200 = 3.5 days` - `= 576 GPU-h` - `(2.6B transactions • 220B tokens)` ## WHY DECODER-ONLY (vs. ENCODER)? - Strong fine-tuning surface — last- ### 00:14:50 # THE BACKBONE ## 1B-parameter decoder-only transformer, sized by Chinchilla ### ARCHITECTURE — 1B-PARAMETER TRANSACTION FOUNDATION MODEL - Input sequence: 1,024 / 2,048 tokens (account history) - Embedding dimension: 2,048 - Transformer stack: 16 decoder blocks - Attention: 32 heads • head dim 64 • masked MHA (Multi-Head Attention) - Per block: MLP (Multi-Layer Perceptron) dim 8,192 • residual • layer-norm - Position: RoPE (Rotary Position Embedding) - Objective: Next-token prediction (Causal + Masked Language Modelling auxiliary) ### SIZING MATH (CHINCHILLA-OPTIMAL) Optimal tokens ≈ **20 × parameters** - DATA TARGET: ≥ **20B tokens** - TRAIN COMPUTE: 6 × 1B × 20B = **1.3 × 10²⁰ FLOPs** - WALL CLOCK: 8 × NVIDIA H200 = **3.5 days** (= 576 GPU-h) (2.6B transactions • 220B tokens) ### WHY DECODER-ONLY (vs. ENCODER)? - Strong fine-tuning surface — last-token pooling works very well. - Causal masking matches reality — we always score the next transaction given history. - Standard infra path — reuses the Large Language Model pre-train tooling. ### 00:15:18 # 1B-parameter decoder-only transformer, sized by Chinchilla ## ARCHITECTURE — 1B-PARAMETER TRANSACTION FOUNDATION MODEL - **Input sequence**: 1,024 / 2,048 tokens (account history) - **Embedding dimension**: 2,048 - **Transformer stack**: 16 decoder blocks - **Attention**: 32 heads • head dim 64 • masked MHA (Multi-Head Attention) - **Per block**: MLP (Multi-Layer Perceptron) dim 8,192 • residual • layer-norm - **Position**: RoPE (Rotary Position Embedding) - **Objective ### 00:15:49 # SYSTEM ARCHITECTURE ## How the Transaction Foundation Model is trained - **Inputs:** - Core banking transactions - Card authorizations - Account & merchant ### 00:16:18 ### SYSTEM ARCHITECTURE # How the Transaction Foundation Model is trained - **Input Data:** - Core banking transactions - Card authorizations - Account & merchant reference data - ### 00:16:51 # SCALING PLAYBOOK ## Three knobs every training stack has to push at once ### FASTER TRAINING *maximize tokens / second* - **Mixed precision** - ### 00:17:18 # Wells Fargo Foundation Model: Application to Fraud Detection at Scale ## ADAPTIVE FINE-TUNING ### From a general transaction model to a fraud classifier ### 01 PRE- ### 00:17:51 # ADAPTIVE FINE-TUNING ## From a general transaction model to a fraud classifier ### 01 PRE-TRAIN - **Transaction Foundation Model** - C