The axiom I was taught in my early days in business was simple but stuck with me: "If you can't measure it, you can't manage it." To reach the potential of generative AI strategies, organizations must measure their success even in their early days to justify ongoing investment. While generative AI promises cost-savings or revenue gains, true measures of generative AI programs will also include innovation as well as human impacts (good and bad) in their scorecards.
Here, I'll cover different ways to measure success—from user satisfaction to the number of AI-generated ideas—and unpack the difference between incremental and exponential definitions of generative AI success. Finally, we'll discuss how to map generative AI projects to OKRs.
Please comment with your views, stories and facepalms—we'd love to hear from you!
For the uninitiated, OKRs (objectives and key results) are a goal-setting framework for organizations, teams, and individuals, focused explicitly on intended outcomes and key drivers (in contrast to KPIs which may measure the performance of a system, but which are not always measuring outcomes). OKRs are intended as an antidote to 'vanity' or 'churn' metrics which measure activity without mapping to meaningful results. For more on OKRs, project management software company Asana has a great guide.
Avoid getting distracted by 'shiny tech'—each incremental use case for generative AI you discover should relate to an OKR your company already values. If not, it's probably not worth investing in. Exponential use cases for generative AI may not map to existing OKRs, but they should at least connect to your company DNA in some way.
Generative AI has the potential to revolutionize how organizations operate. Using sophisticated algorithms and language learning models, businesses can automate or improve operations-related tasks such as customer service and financial analysis. This technology offers numerous advantages over traditional approaches, including cost savings and improved efficiency, by making data and systems 'conversational.' In a survey conducted in the first workweek of January 2023, less than six weeks from ChatGPT’s mass-market release, 29% of Gen Z workers reported that they already used the tool in the workplace.
For example, instead of needing to know complex Excel formulas or delegate inquiries to a data team, an executive could quickly ask questions of their company's data as if they had an entry-level data scientist with them at all times. (The accuracy of this requires a lot of investment in good underlying data, like many generative AI use cases).
The incremental improvements generated by generative AI might go unnoticed initially, but will yield substantial benefits for organizations in the long run. “Could” is the key term here; poorly-implemented solutions might create more friction than benefit—for example, using ChatGPT on its own may not result in brand-aligned tone in messaging and can expose a company to liability through data breaches, inaccurate answers, or poor advice. “Organizations” is the other key term—what’s good for the organization may not always be good for its customers or teams—so they should be considered at least as much as any balance sheet wins. Gartner predicts that by 2025, 30% of outbound messages from corporations will be generated by AI—up from 2% in 2022. Will they meet your customers’ expectations?
Implementing generative AI programs into organizational culture and operations holds potential for improvement across all departments, but is not a substitute for critical thinking and empathy.
However, remember that you can't cost-cut your way to innovation. Will you use your AI-based savings to invest back into your business’ people and innovative R&D? Or just pocket it? Incremental operational improvements using generative AI may seem groundbreaking now, but will soon become a baseline in most industries.
Generative AI has the potential to produce unprecedented levels of innovation in how businesses operate and create new products, services, and experiences. In an era where nearly every company is at least asking about generative AI, it is becoming crucial for businesses to leverage generative AI technology not just to cost-cut, but to be competitive.
From music composition to video game design, generative AI can produce seemingly creative works when well-prompted (though it’s not creative in the same sense as humans). By creating audio or visual content tailored specifically to an individual’s tastes and preferences, organizations can craft experiences that are more enjoyable and engaging than ever before—Spotify is offering everyone their own private DJ, for example. In a sighting from the fashion industry, generative AI can synthesize limitless product photos where it may not have otherwise been financially viable for fashion companies to do photo shoots for every possible product color and size permutation, not to mention with models with a variety of physical appearances and ethnicities. Levi's controversially did this kind of photo synthesis with Lalaland.ai, raising concerns about digital blackface and further discrimination against models who are minorities. Adobe's Firefly suite of AI offerings allows even lay designers to quickly synthesize any number of scenes, allowing for a rapid decrease in the time from idea to mockup. And these tools can be applied in very customer-specific ways, as we've been seeing for a while with chatbots or 'try these glasses on my face' tools.
By following these best practices, organizations can harness the power of generative AI technology for exponential innovation and success.
Generative AI programs have the potential to transform how organizations operate, allowing them to unlock new levels of efficiency and productivity or even create entirely new offerings.
Quantitative measures are probably the most convenient (and demanded). Cost savings are often an important metric for tracking success, of course. Generative AI can help reduce operational costs by automating repetitive tasks such as customer service or support inquiries or laborious editing tasks (turning things from first-person to third-person, for example, or writing customer case studies). It also has the potential to increase sales and revenue through improved customer engagement—provided the AI voice of the company is appropriate to the relationship with the customer. Additionally, generative AI can streamline back-end operations (like IT services, writing documentation, summarizing quantitative data in prose, or translating content).
Qualitatively, customer satisfaction is a key indicator of success when using generative AI programs—as is customer perception of the AI strategy. Investing in properly-managed generative AI solutions promises to pay off through increased customer loyalty and the long-term sustainability of your business model, while poorly-launched approaches can alienate customers and cause real-world harm. Employee satisfaction is another important qualitative measure—do your team members feel empowered by the use of generative AI? Are they happy with its results and able to offload less-satisfying work? Or do they just feel deeply anxious about their jobs? For both customers and team members, good, proactive messaging about what you are or are not going to do with generative AI is essential.
Your generative AI program may be in its early days, but you can still demonstrate empirical progress.
Incremental pilot programs can identify areas that need improvement or optimization to see if generative AI can be helpful. Meanwhile, exponential pilot programs can help identify business model, pricing, or product opportunities and test their feasibility and user/market's interest while identifying potential ethical risks.
Accuracy of tasks generated by AI: How successful were the models used at producing reliable output in comparison with human-generated results?
Time to completion is a key metric when tracking the success of generative AI programs. By automatically completing complex processes that would otherwise take humans significantly longer, businesses can save time and money—ultimately leading to improved customer experience and increased bottom-line performance. But only if the output is of acceptable quality.
The amount of fine-tuning needed for an acceptable result is also important when measuring the success of a generative AI strategy. This metric indicates how much effort is required from developers for the model's output to meet customers' expectations—if too much tweaking is necessary, then this could indicate that the model needs further training or adjustment.
For example, measuring the number of innovative ideas generated or scaled through generative AI can provide valuable insights into whether the investment in this technology has been worthwhile.
Other key indicators in applications of AI to creative or innovation fields include the number and quality of ideas created through generative AI compared with manual efforts, any increases or decreases in time between idea generation and execution, and any harms caused (or prevented) by using generative AI technology instead of a human-only process. Organizations should also survey customers and users to ascertain satisfaction with the results of Generative AI models compared to manual tasks.
Organizations that want generative AI's benefits (or to avoid being disrupted by them) have to have deep digital fluency while also working on their AI fluency. With these two competencies combined, leaders can invest in Generative AI and hold teams accountable—without stifling innovation. Otherwise, the 'incremental immune system' of most organizations will preclude generative AI. Digital Fluency and AI Fluency programs should especially emphasize the importance of robust data capabilities, how exponential initiatives should be managed differently than incremental ones, and realistic expectations (and ethical implications) of today's generative AI. At the same time, organizations should be appointing AI program directors, AI czars or other key point-people to help coordinate their efforts. (May Habib of Writer wrote an on-point post about a proto-job description for an AI program director worth checking out).
Candid conversations about generative AI's impact on strategy, jobs, and customers will build trust and respect between business and tech stakeholders. Go for it—but measure your progress and impact so that you can be proud of what you create.
This is one of a multi-part series on Questions to Answer in AI Strategy. MJ Petroni is a Digital Fluency and AI readiness speaker, author, and accelerator. You can read even more in the Digital Fluency Guide.
.jpeg)
.jpeg)