Your Website Isn't Built for AI

0:00 Kevin White has had one of the most accomplished B2B tech marketing careers that we've had on the show, having led marketing at Segment, Retool, Common Room, and now paving the way for the AI-first customer journey with Scrunch AI. Kevin is not only a leader and innovator, but loves diving deep into the weeds as a proper operator and unbundling bottlenecks to help keep startups moving at the breakneck speed they need to.

0:31 I'm super excited for today. Kevin's going to share how to transform your website for the best possible AI customer experience, not AI for your customers, but building the best experience for an AI bot to interact with your website and brand. Kevin, so stoked to have you here. This is such an exciting topic. Thank you for being on the show and excited to hop in. Yeah, awesome. This is one of the best interests I've got, maybe the best intro I've had so far. I've got to steal that from my LinkedIn bio or something like that. I really appreciate it, and you're far too kind, so hopefully I can live up to that preamble there.

1:14 No worries, no worries. I have no doubts, no doubts. I think, given your career, you've really been at the forefront of modern marketing techniques, approaches, and strategies, and everyone knows it. It's old news by now at this point, but AI is completely making a seismic shift in everything, but I think in some ways that people didn't necessarily expect. So from your perspective, what are the major shifts that you're seeing that are demanding marketers shift their strategy and tactics?

1:52 Yeah, aside from just doing different types of work or creating different types of offers, I feel like this is something we could get into too, but the world of creating white papers and ebooks and all this kind of stuff is like that content is so commoditized that I feel like there's almost like a new way of looking at what kind of offers can we create.

2:13 Can we vibe code something? Can we create an interactive image generation thing? So on that side of things, I'm happy to get into and I've done some of that stuff in the past, but the other thing is, I think we're going to try and talk more about today, is this kind of shift from people and humans consuming your website content and then defaulting to an LLM platform to do their research, to ask questions, to do evaluations of products.

2:41 And so this is kind of left marketers and myself, I was definitely in this camp and still am in some way, is just okay, if all of our traffic of our website is shifting away to this other platform, that seems like both like a scary type of thing that is existential that we need to figure out, but also an opportunity if you can bridge that and do it in the right way.

3:04 And so I really view these LLM platforms like Chat GPT or Quad or Gemini or there's a dozen of them or so, they are almost like the new growth channel for marketing teams specifically to figure out how to understand them and then after you understand them, how to wield them to promote your brand. And so that's one of the reasons why I joined Scrunch is I saw that problem that I was having, I was like, oh, we're seeing traffic coming from these sources, we know that traffic is actually doing like a lot more research behind the scenes.

3:45 This is like human traffic and so there must be something here and how do I figure out even how to tap into this new channel and then I just got curiosity peaked and I just kept on going further and further down and pulling the thread on that and that's what kind of led me to Scrunch too and Scrunch is kind of solving this problem. And so that starts with monitoring, but also gets into how do we intercept that AI bot traffic and serve up a different version of our site and how we make that easier for marketers to create that experience for users and so on and so forth.

4:19 So yeah, I think it's a really cool space. It's very early learning stage and I'm still learning and it's really only taken off in the last year I would say and so I'm definitely not an expert so take everything I say today with a grain of salt, but I'm happy to share what I've learned and seen so far. Yeah, I don't know if there are any experts yet just because it's so new and it's changing so quickly and exactly how the behavior looks like, but I think something that was really interesting to me and I don't know if you have any stats on this or just rough guidance.

4:52 I don't think I realized how much more bot traffic and AI driven bot traffic there is than human traffic. You have a breakdown roughly of like take your run-of-the-mill site. I'm sure it differs a little bit, but like what percentage breakdown is AI traffic versus organic? Yeah, I think most companies are like probably in the 10 to 20% range actually. We have a product actually that I can show. This is kind of a fun thing to show. So let me go to our product if you don't mind and so what it does is it'll intercept that bot traffic and it'll kind of like show you what that bot traffic looks like on your site in real time.

5:39 And so what I can do is kind of give you a sense of like for us at scrunch and because you have to own the website, of course, and so you have to like actually integrate with a CDN, we use CloudFlare to find that bot traffic and then service it up in analytics. And so I can share, sorry, I'm trying to like talk and do things at the same time, kind of like what that looks like to give people a sense of like the volume. To answer the question, you know, I feel like companies are seeing 10 to 20% of traffic from AI sources right now.

6:15 In the future, I think it's going to be more like 50% or maybe like 90% because, you know, that bot traffic is going up and up and up. If you think about what's going on behind the scenes there, it's a human on the other side of things that's prompting an AI and then the AI is going and retrieving that traffic, retrieving your site content and crawling your site and then pulling that response back to the user.

6:37 So in the old world of things in marketing, I was like bot traffic is garbage, like I don't want to know when Google's crawling my site, but now with the types of AI bots that are crawling your site, there could be like actually real intent behind that and oftentimes there is. And so that concept of, you know, this we call it like a retrieval type bot. It's essentially a bot that is taking a real time prompt from a user, going and fetching that data from your website and then pulling it back in the response.

7:07 That's actually someone who's in a buying state. A lot of times they have a lot of intent and so that traffic is actually going to become super important. And so this is a look at our dashboard here. It's using our scrunch website because again, we are the only ones that have first party data that can like actually surface this. But you could do it for any site. You just have to integrate with your CDN like Cloudflare, Purcell, Akamai, WordPress, all these things that we integrate with and you can kind of like see this bot traffic on your own site.

7:37 So we break it down by a few things like retrieval bots. These are the bots that are like doing that rag type of search, a user prompt something, the bottle go fetch something from your site and pull it back. There's like the training bots that you can see, which are more like long term type of like we're going to use this, we're going to crawl this site as an LLM and then we're going to use it to train our model. Those are a little bit more like what the initial version of chat GBT and others were like. And then there's like your typical like Google index search bots.

8:06 And so you can see at the top bot agents visiting our site here, you can see the different pages that are being crawled the most by these bots. And then what's even interesting in our conversation here in the last since like 11 o'clock, you can see chat GBT is hitting up our site. Cloud training is hitting up our site and you can see the type of bot and the pages that are that are being visited. So someone on this one could have said, you know, what is scrunch is with scrunch is funding. What is stretching scrunch is pricing model and this chat GBT retrieval bot will go hit our site and then pull that answer back to the user.

8:38 So this is almost 100% guaranteed a user chatting with chat GBT and then that chat GBT traffic is visiting our site on behalf of the user and so the user is not actually coming to our site. So we think I think in the future like this is what we're going to be building for markets. We're going to be building a site experience that works on the human end when someone's visiting your site and it looks nice. But then also we're feeding this AI traffic the right kind of content that it wants to surface up the right information to the people behind the prompt.

9:08 Yeah, and I think I think that's an interesting part that a lot of people don't think of it was a nuance for me that hey, that bot traffic isn't actually bot traffic. It's it's people behind the bot. So they're sending their executive assistant to go check out your website and get the information that they need, but there's an actual person who's receiving that information. So that's actually I would argue in a lot of ways probably some of the highest intent traffic that you might be getting because you might get random people coming to your website on accident.

9:41 But if somebody is going into an L I'm asking a very specific question and your website has relevant information for that question and then you're getting positioned in front of that person, that's really, really important interaction. Yeah, the behavior that we kind of infer and some data that we've seen there's been studies on this where if you're getting human traffic from an LM like they click on the citation and actually get to your sites.

10:09 That traffic is actually converting a lot higher and higher has usually a higher value with the visit compared to like, you know, organic or direct traffic. And if you think about the reason why is it usually the person is prompting the GPT or the LM with many like a long stream of questions like compare this product versus this other product. I want to learn about this. I have this problem. I have this pain point and they're getting a feed of information. And then once they get to a point where it's like, oh, I want to make a decision now.

10:42 That's when they would click the link and come over to your site and then convert. And so we you kind of see that behavior play out with your traffic from LMS. But it's even like but by capturing that bot traffic, you get a sense of what's happening before that as the bots are doing the work with as as the person's chatting, if that makes sense. Yeah, totally makes sense. And I know it's early days. So I know there might not be a full formula for this. And I know the LMS may behave a little bit differently in the near future, who knows.

11:16 But current state, are there any practical things a company can do to make sure that their website is set up to accommodate the bot traffic and give it the information it needs and do it in a way that gives them what they want efficiently and passes it back to the person that's working with it? What are some practical things people can do? Yeah, well, the one thing to start is just starting to measure this traffic in general. And so whether you're going to measure it through Google Analytics or product like scrunch, I feel like just getting a baseline of, you know, this is the bot traffic that's visiting my site.

11:58 And like, why and trying to interpret like, why is this bot traffic there? And what is it doing? And like, what is the intent behind it gives you like a baseline and then you can kind of like measure like, oh, is our, our, is our bot traffic going up over time? Is it going up for the right, the right types of retrieval bot versus a training bot, etc. So just like baseline measurement, which is like what I would say 90% of people like don't even do right now is probably a good starting point.

12:26 From there, it's kind of like there are the bots have a way of crawling and indexing a site. And so they really speak the language like the lingua franca for for AI bots is markdown and JSON. So if you can optimize your site to feed this AI traffic with that type of structural like technical content, also like schema markup, like, this is an author post, this is a, this is a definition whatever like Google has this whole like, you know, this is the schema that you can use for your content.

13:03 AI is really good at like consuming content, it wants to like consume content in that way. And so some broad strokes things are like if you have a bunch of like fancy JavaScript and animations and stuff like that on your site. AI is not really good at parsing and understanding that JavaScript yet, it might get there but that is one thing that, you know, you might want to look out for other things that we've seen kind of AI get trapped tricked up with is the structure of like a pricing page is a really

13:31 good example where there's multiple columns, and there's different pricing structures and different features for like four or five different plans, for example, and if you look at the code behind it because the AI is not looking at the site, like, like a human would it's looking at the back end code. It's oftentimes like structured in a way where it mixes up the pricing from the pro plan to the features of the enterprise plan and all this kind of stuff. That is like a good example of, you know, structuring your content in the right way for AI to consume it by putting in like nice markdown nice format helps it understand what the

14:05 way it should be referencing and servicing that information back to the user behind the prompts. I don't want to plug scrunch too much but but that is like the product that we are building as well which is not only intercepting this AI site traffic is in giving you analytics behind it, but servicing up a separate version of your website a parallel version of your website that includes things like the AI needs like markdown JSON clean format but then also maybe additional content like more FAQs about this topic or different content to supplement what that page is intent is and so

14:38 that kind of separates like you have this nice site that looks great for humans but then also like a separate parallel site for AI which is also super important traffic. And so we see that as kind of like the future or the way of things happening with marketing is to build this kind of site that is that is optimized for AI to consume and that's how you communicate with this channel. Yeah, and I'm wondering, in the work you've done with customers, do you see any specific results so once you start to optimize or the LMS to receive the information well teed up to the people there that are getting prompted.

15:20 How, what type of tip and performance you typically see once they've implemented some of these practices. And this is the part where like heavily caveat things is that it's definitely correlation, but it's, it's impossible to know the causation because the AI like LMS platforms don't open up like this is how we score algorithm on things. But we, when we, when we roll out this product, you can see that the content that you, the first thing is just showing that it works. So it's like, we fed AI this content, your site was actually formatted in this way before, and after the AI is now

16:03 servicing an accurate representation of that pricing page, that FAQ page that whatever so that's like the first thing is just getting the getting the accuracy of the content that you want to communicate that over to the AI in the right way. And then the other thing that we see is definitely performance improvements in terms of like traffic and prompt visibility and citations and stuff like that that's actually the first part of our product is just baseline monitoring your AI presence comparing it versus competitors

16:32 and different stages of the funnel and all these other sorts of dimensions, and then like getting insights to optimize that but then the third part is this like, you know, build a site that's built for AI consumption. But we do see when best practices around, I hate the word best practices shouldn't be used best practices but that's okay. When people are taking those insights and implementing them, they see, you know, one of our customers has like

16:59 this X performance improvements and conversions called the run pod. There's a customer story up on our site. And I would say, this is, it's, I put a huge caveat on this because I'm not going to say like overnight, you're going to get 6x more traffic and conversions and all this kind of stuff. Sure. It's a rising tide lifts all boats so you're also getting more people using AI and more traffic from AI in general, but we do see heavy correlation with if you implement these things, like,

17:28 using authors on your site, chunking out content, putting stats that are credible, like those types of things tend to help the LLMs surface up the right kind of content to the end user. And as a way of doing that, then you tend to see a correlation with higher traffic and higher conversion rates. Yeah, makes a ton of sense. And I know I, early days, who knows what best practices right now. I think there's a currently this is what some people are doing and I think a lot of people aren't even aware of some of this stuff going on.

18:01 There is, there is one thing, sorry, there is one thing that I caution kind of against, which is a lot of, a lot of folks out there will say, hey, you need to find untapped content opportunities and then use AI to generate content, which, to me, ends up you create these workflows that like generate a bunch of pages like anyone can create a page nowadays with just a prompt and it takes zero time and zero effort.

18:31 And so I feel like that type of content is AI sloth and it might work in the near term, but I feel like in the long term, it's going to be hit with a Panda like update and like all that traffic is going to go away. So maybe now's the time to do lean into that arbitrage if you're open to the risk. But I feel like over time, like that strategy is going to be a losing strategy and the LMS know if something's generated by an LMS. So at some point, I think they'll turn off any kind of visibility or any kind of traffic to that if it's not actually helping servicing the user, you know.

19:07 Yeah, I think that makes a ton of sense, which I'm excited about. I'm excited for, hey, people who have real content to share, organic stuff to share, actual insights. I mean, I've seen it a million times. You go on somebody's website and they have hundreds of articles. Oscillary? Yeah, a bunch of definitions of things, yeah. It's like, come on, nobody wrote these and nobody put those together. But in your opinion, there's such an industry around SEO and, you know, there's so many companies and tools and everything just like built just to do that.

19:44 Do you think we're going to see the same thing or what it's been branded as, AEO right now? Or do you think the nature of the LMS themselves will kind of make it difficult to gain the system? Yeah. Interesting question. I feel like we're already starting to see all these types of AEO, GEO companies spin up. I mean, Scrunch certainly has dozens, if not maybe hundreds of competitors already. Some of it's the incumbents like SEMrush and Ahrefs that are building something similar and they're big in the SEO space. And then there's other like startups that spin up, you know, almost every day, every week.

20:27 So I actually think that that market of monitoring AI traffic, well, I think our product is, you know, the best one out there. And I'm biased. I do think that that's going to be a commoditized feature. And where we'll start to see diversions in the market is the different philosophies around like, oh, we have this philosophy of changing the content of your website to service this AI bot traffic. But others might have a philosophy around, you know, building content workflows or something like that or some other divergent market differentiator there.

21:07 So the market is definitely shaping out. It's still early days. I feel like all this traffic from AI started coming more online when chat GPT introduced citations and sources. And that seemed to be like the January 2025 period. And we started seeing like more and more traffic over time. And then people are like, oh, I need to measure this traffic. And then I need a tool for that. And so that's how the market kind of got started. But I feel like over time, it's going to evolve quite a bit. And there will be more science around it, I would imagine. The other question around is, are the AI platforms just going to pull up the ladder on us?

21:50 Maybe, I mean, that's totally possible. They could just make a super app and there's nothing else that exists. I don't think that that plays out in the market so far. Like there's Facebook, there's Google, there's Amazon, and there's also Shopify. There's like so many different other alternatives to things out there that I feel like there's going to be a market across the board and specializing. I don't think OpenAI is going to like just be the only company that exists, if that makes sense.

22:23 Oh, it totally makes sense. Well, it makes sense. I agree. I think that's probably a hyperbolic view of how it might end up. But I think, I'm just curious how that industry is going to shape up and how much you can, I don't want to call it gaming because I know some of it's just like, hey, we're going to create, put out information that is helpful. But I'm just wondering how much of it you can game versus what the SEO industry seems to have done. What I actually really like about some of the differences that I'm seeing that were, you know, again, this is all a black box.

23:04 So but like some of the correlation that we're seeing is with the way to optimize for AI traffic is actually to focus a little bit more on like bottom funnel type of queries, like long prompts that answer like very specific questions. And if I, you know, I think there's a world where, you know, you're the only company out there that can answer this question, or there's maybe like two or three out there and like no one's going to try and game that prompt. And so you, you can create content around this like very specific thing and like AI will find people who are interested in that and then service your content to them.

23:45 I feel like we're seeing more of that than than compared to something like SEO, which is like rank for this keyword. It has all this awareness. It's a super competitive game. And it's like really, it's either expensive to pay Google the vig to show up there, or like you have to have like a content army and backlink army to like rank number one, you know, so I feel like it is still early days. You know, it's like, you know, over time, it might become harder and harder as people figure out, you know, new opportunities and it gets known and spread and marketers ruin everything.

24:15 But right now it seems like focusing on what matters for that bottom of funnel like answering users like true pain points and making that specific to something like your brand or your, your thought leaders can only answer like that seems to be working actually pretty well. I think these insights are incredible. I'm learning a ton right now. I'm just curious. Is there anything else as a business we can be doing to put our best foot forward with these LMS and just give them the information they need.

24:48 Yeah, I'm a big fan of instead of reading something or digesting some content or listening to this podcast to go out there and actually try it yourself like getting your hands on, especially in a new market like this. There's a new channel like the best way to learn it is to go out there and do it versus listen to people like me talk about it conceptually. That said, I do feel like there is kind of like a minimum thing you can do, which is just to start measuring the stuff, measuring how your brand's showing up and prompts measuring where your competitors are showing up if you're being cited or not.

25:27 Like that baseline, at least give it's kind of like Google analytics in their list. You need a baseline to understand what's happening with analytics and then you can kind of like make inferences and insights and decisions based off of that. And so I would encourage people to look at look into a you can do this manually. Actually, you can like put these prompts into a spreadsheet. You can prompt the LN models yourself. You can probably build a custom GPT or something like do all this for you.

25:52 I would say that's probably not a good use of time when there's like a, you know, credit card transaction type of product like scrunch out there where you can just, you know, set this up and we can do all that mess for you. But try one of these products. I mean, I hope everyone here will give scrunch or try to, but like, you know, there's plenty of other products out there to just measure this from a baseline. And then from there you'll I guarantee you'll start to get ideas like, oh, we're seeing this this corner of prompts aren't being aren't being addressed the right way.

26:25 We can create content around that or we can like change some of our site content to address that and then you'll start building and learning over time. And so that's my advice is start measuring it, but then also start doing and start trying things because that's where you'll get the most learning from. Yeah, I think that's great advice. Just get the feedback we've going now start iterating moving forward. And I think to your point, yes, hands on best way to learn things. I may arguably the only way to learn things. Well, I'm I'm curious. I'd love for you to share just I think you've had such an interesting career in marketing.

27:01 And I'd love if you could just share how you ended up at scrunch and kind of the experiences that stacked along the way to get you to where you are today. And one thing I always ask if our guests can share if there's anything you attribute your success to or anything that you think someone could copy and emulate for themselves. Yeah, yeah. We'll see. That's a tough, tough, tough thing to answer. I feel like I'm like reflecting back introspecting on my my career. No, I think, you know, the thing that got me to where I am is curiosity.

27:45 And so what I mean by that is when I first started in SaaS marketing, essentially I this was back in the day where I I joined the SaaS companies called Gigia. Actually, the the CEO of Gigia is now an investor in scrunch. And one of the reasons why I joined scrunch. Well, that's a different story. That's a shout out, Patrick Salier, if you're listening. But like when I joined this company, I was very junior, not, you know, not knowing how SaaS and PLG and all this kind of stuff works.

28:19 They're just like, hey, we bought this thing called Marketo, like go figure it out. And so that essentially just got my curiosity building and learning kind of learning by doing mentality going or just like, I need to figure this thing out. I need to understand how to generate revenue, how to like get people interested. And from there, like, I kind of just that got me into more of like a growth mindset where it's like, oh, we built out this infrastructure for lifecycle marketing. How do we get people on the top of the funnel? How do we get them to engage with this stuff?

28:51 And so, you know, it just by by setting something up, you kind of like get this curiosity of like, how do I go and like learn how to achieve this objective that we want to do? And then I feel like that leads a lot more to like a growth type of mindset, like a growth marketing type of mindset, which is, hey, we need it, we need to solve this thing. I we need to do both like the analytical and the creative part of it. So like the creative part is like, oh, what offer can we put together?

29:22 What channels can we reach to get to this audience? But then also the the infrastructure and data driven side of things like when someone actually does this action, you know, how do we report on that? What insights can we learn from it? Like what data science can we apply to that? And like merging those two things like tend to be a really good combination for a growth marketer. And growth marketer is a really good stepping stone to leading all of marketing, I would say, because I mean, everyone needs growth and distribution.

29:53 Otherwise, you don't have a business. So I would say that has been what's got me to where I am in my career to leading growth at segments and then also, you know, leading marketing at retool and then common room and then now scrunch. So, yeah, hopefully that somewhat answers the question. No, I think it makes a ton of sense. And I think taking that growth marketer path as the path to, you know, running the whole show ahead of marketing CMO, I think I think you're right. Closer you can get to the revenue performance and then use everything else as a tool brand and your channels and everything else to manage and techniques.

30:30 But being can I add one other piece of career advice? Yeah, to this as well. So on paper, that career trajectory looks really great. But be careful what you wish for because I got myself to a point where I was leading a marketing team, I was really ambitious and what your day to day looks like from where you're starting to grow to where like you're leading a team of like 30 people is very different.

30:59 And so I ended up at common room actually taking like a step back and taking more of a super icy role because my day to day was just filled with board updates, internal marketing, dealing with drama from the SDR team, like all this kind of stuff that was just like killing my joy.

31:16 And so if that's there are people who love doing that stuff. And if you aspire to do that, like do it in great. But also, there's another path for being an IC. And I feel like especially nowadays with AI companies being small and being pretty flat, like that type of role is actually something that, you know, if I was starting early, I might go in more in that direction, knowing what I know now. Yeah, I think that's great. And you may have to go through that experience to even know because it looks so alluring, like, when you're early in your career and, you know, wanting to be in a position to make an impact and be a leader.

31:51 But to your point, the day to day, it's pretty treacherous and mundane and kind of boring sometimes. So, and it pulls you away from some of the like newer, exciting hands on work that you could be doing in that temporal team. Yeah, and the pay is not all that much more for the stress. So, maybe equity wise it is. But yeah, anyway, different topic. No, I think that's really, that's sage advice. And I appreciate you sharing that because I think most people have in their heads, like, hey, how do I rocket ship to the C-suite? But are you sure that's actually what you want to do?

32:27 And I do think there are more opportunities now because the organizations are relatively flat, where even if you have a C title, like, oftentimes, you still have to roll up your sleeves and do stuff. So, I think that's a good thing. I think most of us get attracted to the work, not necessarily just professional management, and especially board management and things like that. So, I think it's a great path, if you can carve it out. Yeah, I feel like I wouldn't be able to give advice if I wasn't hands on with things. And so, despite driving some of my team members crazy, and I'm a team of one right now at Scrunch, so I can drive myself crazy.

33:05 Despite that, I still feel like I must be hands on with something. And for the people who are in that, you know, path of destruction, who are working with me on that thing, I feel sorry for them. But I'll pull back and bother another team member at some point as I build my game. So, yeah. Amazing, amazing. Well, Kevin, this has been awesome. I think you really illuminated this stuff for me where, hey, a lot of the traffic already 10, 20% of the traffic is really guided by LLMs retrieving information, teeing it up to people who are prompting them to get the information they need.

33:43 That's only going to increase over time. I love the practical recommendations and tactics that you gave to help put your best foot forward and just get started measuring so you know how much of an impact it has on your business. Scrunch AI, your technology, your platform looks incredible. So people can leverage that to start getting a baseline. And then who knows what the future holds. But I trust someone like you is going to figure it out and move with the waves as they come. And thank you so much for sharing everything you did today. Of course, yeah, thanks for having me. It was a blast. I had a good time. Awesome. Thanks, Kevin.