Buyers need keep information to feel confident about touring a home.
While Redfin’s listing pages provide details, buyers often have additional questions.
The current Ask a Question feature’s slow response times can discourage potential buyers, causing them to miss opportunities in competitive markets.
Challenges
Redfin is currently gaining market share from its close competitors but is struggling to gain traction on the increased traffic as lead generation rates stay stagnant. These top-of-funnel gains cannot be negatively impacted by any mid-funnel solution; any solution we build would sit between a listing view and a tour, introducing roadblocks and delays in the buyer’s journey.
Because of where this solution would sit and what teams it might require resources from, the team must navigate the following:
- Complex organizational structures among support staff, agents and partner agents
- Non-standardized “behind the scenes” data is valuable, but messy
- Competing priorities in Machine Learning teams
- AI technologies are untested with existing frameworks and the training of LLMs can be intensive
- Ethical and Fair Housing concerns around generative content
- Continuous communication and integrating with other team’s work and resources
- Top-down directives and expectations
The Work
- Fully understand the impact (+/-) of AAQ on customer decision making.
- Explore secondary impacts from Tour Scheduling and common communication patterns between buyer, support staff and agents.
- Align on a strategy and solution that increases a potential buyer’s education and confidence on a single listing.
- Align on a solution that integrates the company’s interest in AI and leveraging “behind the scenes” data.
- Design and build a solution that positively impacts and drives more conversations with Redfin and a smoother handoff between digital experience and agents.
Current State
Home buyers are 27% more likely to tour a home after they have their questions answered through AAQ. ~80% of questions asked are about the listing, while ~11% of questions are about Redfin services, which on average take more time to resolve.
Qualified Leads (most likely to tour)
Avg. Time on Listing: 4.66 min
Avg. Time to AAQ: 2.4 min
Avg. AAQ Response Time: ~15-30 min
Avg Time to Tour: ~3-4.5 min
Crafting and Communicating a Vision
“Empowering every homebuyer with intelligent, personalized guidance through seamless AI-driven interactions, extending Redfin’s trusted service to every step of the journey.”
This could be supported by key service principles:
1. Instant & Intelligent
- Questions answered in real-time
- Responses that combine property data, agent notes, unpublished listing details with market insights
- Smart enough to know when human expertise is needed
2. Seamless Progress
- Every interaction moves the buyer forward
- Natural transition from questions to tours
- Context preserved across the journey
- Questions asked build agent’s buyer awarness
3. Confidence Building
- Comprehensive information delivered conversationally
- Clear path to human expertise when needed
- Transparent about what’s known vs. what needs agent input
4. Always Available
- 24/7 response capability
- Consistent quality of information
- Proactive surfacing of relevant details
In Q4 of 2022, Redfin hosted it’s annual corporate gathering, Redferno, where the leadership team outlined the company’s core business vision for the next year, becoming a “Destination for Service”. As a leader in the real estate space, Redfin saw a unique market opportunity in our brokerage, our people and agents. Redfin wasn’t just an online service, but a pathway to the people that drive the business and its unique services.
This vision supports Redfin’s core business through:
1. Business Impact
- Reduce cost-per-conversion by automating initial Q&A
- Increase tour conversion rates through faster, more confident decision-making
- Free up agents to focus on high-value activities like closing deals
- Capture more market share in competitive situations where speed matters
2. Technology Leadership
- Position Redfin as the most responsive platform in real estate
- Leverage AI/ML capabilities to create a defensible advantage
- Build on existing data advantages from Redfin’s transaction history
3. Customer Experience Differentiation
- Transform “asking a question” from a potential drop-off point to a conversion driver
- Create a more empowering self-service experience
- Maintain the human touch where it matters most
4. Platform Growth
- Generate more behavioral data to improve AI responses
- Create opportunities for cross-selling other Redfin services
- Build stronger customer relationships through consistent availability
Communicating this vision to the executive team was tricky in that we know, in order to get full buy in from Glenn Kelman, our CEO, we had to build a little show-and-tell presentation that included visual mockups of our where this vision might take us.
Leadership note: Early mockups can bias the design process before we have proper data and research. However, I created initial visuals to spark discussion and excitement around possibilities, while making it clear that the final solution would be shaped by team collaboration and validated insights.
I chose to share 7 different flows that describe typical user behavior (using AAQ as a model) augmented by AI generated agent content leading up to a conversation with Ask Redfin, and finally a hand-off to a human.
Developing a Design Strategy
“Speed to Confidence” is critical because it directly addresses our core problem: potential buyers abandoning their journey due to delayed responses. This pillar ensures we’re not just making responses faster, but actively building buyer confidence through every interaction. By combining AI-powered instant answers with progressive information disclosure, we help buyers make informed decisions quickly in competitive markets.
“Smart Escalation” recognizes that while automation is valuable, human expertise remains crucial in real estate. This pillar ensures we maintain the high-touch aspects of Redfin’s service while improving efficiency. It’s about striking the right balance – using AI to handle routine queries while seamlessly transitioning to agent expertise for complex situations or high-intent buyers.
“Data-Driven Personalization” creates a virtuous cycle between user interactions and service improvement. Each conversation becomes an opportunity to learn and refine our responses, while also allowing us to proactively surface relevant information based on patterns we observe. This pillar helps build a defensible advantage through accumulated learning.
Implementation Approach
The team had to quickly put together a project plan, I worked with our Product lead and designers to help build out a phased design work approach where we’d focus on the critical components first, and then build on top.
Measuring Success
We identified a set of metrics we hope to positively influence after launch – these would be adjusted based on technical capability:
Buyer Impact
- Time to tour scheduling
- Question-to-tour conversion rate
- Buyer satisfaction scores
- Suggestion effectiveness rate
- Time-in-conversation
- Questions Asked
- Listing
- General buying
- Services
- Return visit rates
- Frustration signals
- Exits
- Conversations
- Unanswered questions
Business Impact
- Controllable Close Rate
- Support Staff Q&A cost
- Cost per tour scheduled
- Agent efficiency metrics
- Market share in competitive situations
- Cross-selling conversion rates
- Other buyside scheduled events
- Consultations
Research Planning
Conducting Initial User Research
Leadership note: Redfin had laid off its entire research team early in my time there. I took it upon myself to make sure that this work was supported and continuously fed with important customer and data insights that I would continue to gather, synthesize and share during the course of design.
First, we identified key hypotheses to test:
- Users drop off due to slow response times
- Natural conversation would encourage more engagement
- Early guidance could improve tour conversion
- AI could effectively handle common property questions
Then, we mapped research methods to business impact:
Immediate Optimization
- Analyzed current AAQ data to identify patterns
- Reviewed agent feedback logs for common pain points
- Studied drop-off points in the current journey
Foundational Understanding
- Conducted user interviews about their home search process
- Shadowed agents handling buyer questions
- Examined successful vs unsuccessful buyer journeys
Solution Validation
- Created conversation flow prototypes
- Tested AI response quality with real user questions
- Evaluated handoff triggers with agents
We prioritized research activities based on:
- Impact on key metrics (tour conversion, response time)
- Technical feasibility questions
- Resource requirements
- Timeline constraints
Initial Findings
We ended up talking to 20 potential home buyers, sourced from our in-app opt-in and from the active agent pipeline (buyers who were currently in the tour and offer process).
We also spoke with several Redfin agents, partner agents and support staff.
I’ve condensed some of our findings below:
Agent Value
Agents translate desires into real search criteria and provide crucial guidance through the home buying process. They adapt search strategies based on feedback and changing needs.
Flexible Starting Points
Buyers often start with unclear needs that require interpretation. Agents help uncover true requirements and suggest alternative solutions, like recommending ADUs instead of basements for in-law housing.
Self-Service Preference
Modern users prefer self-service options, especially in real estate where they may feel anxious about appearing uninformed.
Conversation Power
Natural conversations engage users better than traditional UI elements, encouraging more interaction with Redfin.
Smart Search Translation
Ask Redfin digs deeper into user needs by asking “why” and helps refine search criteria based on true priorities.
Guided Journey
Instead of overwhelming users with maps and filters, Ask Redfin provides personalized guidance through the buying process, including education on key terms and considerations.
Pre-Tour Confidence
Proactively surfaces relevant home features based on known user preferences before tours are scheduled.
Buyer Confidence
Provides balanced, honest answers about listings, including potential concerns about price and location.
Agent Preparation
Helps agents make better first impressions by summarizing user goals and concerns before initial contact.
User Learning
Gathers user information naturally through conversation rather than formal onboarding, making the process less intimidating.
Broad Application
Solution should serves users across their entire real estate journey, from rentals to off-market properties.
Working with Vellum
Vellum is a prompt engineering tool for experimentation, evaluation, deployment, monitoring, and collaboration.
As the design work was well underway, leads in design, product and engineering took to Vellum to help build and test our prompts to ensure accurate, valuable and ethical responses from the fledgling virtual assistant.
We spent hours testing and validating hundreds (if not a couple thousand) different prompts and variations to ensure that we were launching a fair and compliant real estate AI assistant.
Challenge Real estate conversations involve sensitive decisions and strict legal requirements:
- Fair Housing Act compliance needed
- Licensing restrictions for real estate advice
- High stakes decisions for users
Strategic Approach
- Clear Scope Definition
- Focus on factual property information only
- Avoid advice-giving functionality
- Strict boundaries around agent-only services
- Test-First Development
- Built comprehensive evaluation framework
- Created thousands of test cases
- Used Vellum’s evaluation suite for systematic testing
Implementation
The team implemented a comprehensive testing and validation process that went far beyond typical AI development practices. By leveraging Vellum’s evaluation suite, we were able to systematically test the AI assistant against thousands of scenarios, ranging from basic property questions to complex situations that might approach legal boundaries. Each test cycle provided insights that helped refine the AI’s responses, ensuring they remained factual and compliant while still being helpful to users.
Results
The careful development approach paid off, resulting in an AI assistant that successfully navigated the complex requirements of real estate conversations. The system demonstrated consistent ability to provide helpful property information while staying within defined legal and ethical boundaries. The extensive testing framework proved valuable not just for the initial launch but also for ongoing monitoring and improvements, ensuring the assistant maintained high standards of compliance and accuracy over time.
Key learnings about working with and training an LLM
Through this project, we discovered that proactive compliance testing was not just a legal necessity but a crucial foundation for building user trust – Redfin’s brand is built on trust in the industry. The systematic evaluation approach, while time-intensive, proved far more reliable than relying on intuition or limited testing. Perhaps most importantly, we learned how to effectively balance user needs with legal requirements, creating a system that could be genuinely helpful while remaining firmly within appropriate boundaries.
Supporting the Work
As a Senior Manager, much of my involvement is seen through supportive measures meant to clearly establish roles within the design work and ensuring healthy collaboration between all departments and stakeholders.
Team Leadership & Alignment
Activities:
- Weekly design reviews focused on conversation flows
- Bi-weekly cross-functional critiques
- Monthly design strategy check-ins with product and engineering
Scenarios I managed:
- Designer struggling with AI response patterns within conversational UI tool – organize pairing sessions with product, AI team or reach out to vendor for support.
- Conflicting feedback from stakeholders – facilitate alignment discussions where we’d go over feedback directly with stakeholder – limiting to 15 minutes.
- Team feeling overwhelmed by timelines and resource constraints – restructure workload and priorities
Process & Quality
Activities:
- Define conversation design patterns as guided by Blueprint
- Create testing protocol for AI responses with Product leaders and legal
- Regular Blueprint compliance checks for all new design components providing actional feedback to designers
Scenarios I managed:
- AI responses missing context – work with AI team on data integration
- Agents concerned about handoffs – prototype new transition flows that manage concerns
- Technical limitations affecting design – explore creative alternatives using either existing designs or functional patterns
- Teams misaligned on design priorities – return to prioritization framework and make sure that designers are conducting their own check-ins and updating tickets in Jira with correct statuses
Cross-functional Partnerships
Activities:
- Joint planning sessions with AI/ML team
- Regular check-ins and working sessions with Brand Marketing teams
- Agent feedback working sessions
- Engineering feasibility reviews during early design work
Scenarios I managed:
- Inconsistent AI responses – establish tone/voice guidelines
- AI responses trigger legal and ethical concerns – establish immediate lines of communication with legal team via Slack “office hours”
- Performance concerns on listing pages – coordinate with engineering on optimization and reach out to other teams who may be working on connected features
- Design system conflicts (rarely) – organize Blueprint integration sprints
Key Design Decisions
As the design team worked through the flows, there were a couple of questions around how we might introduce agent handoff sooner and make the handoff feel a little more seamless without disrupting the user’s session with the assistant , I had suggested a couple of ways to do that.
Quick Prompts
Allowing the user to select a pre-defined response that the bot automatically presents to the user based on the current conversation context, especially useful when:
- The bot cannot accurately answer the question with confidence
- The bot is asked a question unrelated to the current listing
- When the bot understands that the user wants to tour the home
The user asks a question that the bot has been trained not to answer, its subjectivity is too high, and may violate Fair Housing guidelines.
The bot responds with a soft redirection, asking the user if they want to tour the home or meet with Leah (the agent) to get more detailed answers.
Quick prompts offer a quick selection of suggested times for the user to schedule a tour with Leah – these suggestions are pulled directly from our integration with Agent Availability and Scheduling.
Non-disruptive Agent Hand-off
Another challenge was around how we might handle moving the user from a conversation with the bot into a conversation with one or more humans.
I had proposed a solution where we could introduce a new chat view, preserving the integrity of the AI assisted experience which were conversations built around a single listing. When users moved from listing to listing, they would only see conversations about that listing – it could be incredibly confusing to see a conversation thread about multiple homes with no clear way to isolate each listing.
After selecting a meeting time, users receive a confirmation and a clickable link to connect with Leah.
The user is taken to separate conversation thread, where once again a tour is confirmed and a person from Leah’s team responds in real time.
All conversations can be accessed via a center stage Inbox selection in the apps main navigation.
Impact & Results
Ask Redfin has shown promising engagement metrics and is effectively driving demand, though opportunities remain to expand its impact. While users who engage with the feature are highly valuable and show strong intent, the relatively small daily user base limits its overall influence. The conversational format has proven significantly more engaging than the previous Ask-A-Question solution, leading to longer interactions and higher submission rates. However, lower closing rates compared to other conversion channels suggest an opportunity to better capitalize on user intent, particularly since these users demonstrate a higher likelihood to purchase. The feature’s success in question resolution and positive feedback provides a strong foundation for growth, especially given the documented user need for better home discovery support.
Current Performance
- 60% question resolution rate (AAQ @ ~80%)
- 67% positive feedback (AAQ @ ~50%)
- 14% 30-day return rate
Growth & Engagement
Compared to listings not supported by Ask Redfin
- +24% contact growth since national launch
- 40% increase in question submission
- 75% longer conversations than previous solution
- 2-3% increase in detail page views for users
User Behavior
- Only 0.08% of daily active users (600 users/day)
- Users 10% more likely to buy vs. tour requesters
- Lower closing rates than other conversion channels
- High-value users with above-average price points
- Limited use for non-Q&A purposes (0.5% home search questions)
- Engagement comparable to tours/offers users
- Performance better than engagement features, worse than conversion features
After the release of Ask Redfin, there was still considerable gap that we feel AI assisted features could bridge.
- 51% of users struggle to find matching homes
- 20% struggle with decision-making
Final Thoughts
As the design leader for Ask Redfin, this project reinforced the power of conversational AI while highlighting crucial opportunities for evolution. While we successfully launched a product that improved engagement and question resolution, the real learning came from understanding how AI could serve broader user needs throughout the home buying journey.
The most significant insight wasn’t just about making a better chatbot—it was about recognizing that AI’s potential extends far beyond conversation. Our users were telling us, through their behaviors and feedback, that they needed an intelligent assistant that could surface relevant information proactively, guide decision-making contextually, and provide support across multiple touchpoints.
Three key opportunities emerged that will shape future development:
First, AI needs to break free from the chat interface. By integrating AI capabilities directly into listing pages, search results, and other key interfaces, we can deliver intelligent insights where users naturally interact with our platform.
Second, we need to expand our vision of AI assistance across the entire home buying journey. From early-stage calculators to budgeting tools and tour scheduling, there are numerous opportunities to reduce friction and increase confidence through AI-powered guidance.
Finally, we must leverage our growing understanding of user needs to create more personalized, journey-aware experiences. Each interaction is an opportunity to learn and adapt, making subsequent interactions more relevant and valuable.
These learnings have fundamentally shaped my perspective on how we should approach AI integration at Redfin—not as a standalone feature, but as an intelligent layer that enhances every aspect of the user experience and the entire customer journey.
