{"id":373260,"date":"2024-05-21T05:32:07","date_gmt":"2024-05-21T05:32:07","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=373260"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=373260","title":{"rendered":"How to speed up Trendwatching with AI<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\n

Problem <\/strong><\/p>\n

Trendwatching is a powerful tool for driving strategic innovations. It helps to discover new te\u0441hnologies, business models and products that may be used for idea generation and technology transfer. It is a powerful tool for product managers<\/u>, business stream managers<\/u>, top managers<\/u> and «strategists» and is mostly used on a regular basis.<\/p>\n

\"Flow<\/p>\n
Flow Of Ideas For Innovations<\/figcaption><\/div>\n<\/figure>\n

Most trendwatchers use meta analysis of ready-made reviews and reports. However, an objective and undistorted approach means that you also have to scan the emerging «signals»: news, startups, venture deals, academic articles and patents day by day. In this case you have to analyze the huge amount of text data<\/u>…Big Data. Indeed 50 million startups are created<\/a> globally each year, 4 mln per month. Each year, over 2 million new research articles are published<\/a> in more than 30,000 peer-reviewed journals across all fields of study, 167,000 per month. How to succeed in trendwatching with limited resources and tight deadlines? <\/p>\n

Solution<\/strong><\/p>\n

Natural Language Processing (NLP) and AI are definitely what we need to cope with Big Text Data in time. Entity Recognition<\/u> and Text summarization<\/u>: these are the two tasks that require some analysis. Let’s start with the first one. Let’s imagine that we want to put perspective technologies that appeared over the past year in a certain business or technology sector on our «TechRadar». Let’s focus on «AI» for example. <\/p>\n

Pipeline: Tech Trends in AI<\/strong><\/p>\n

    \n
  1. \n

    Download startup descriptions (2023) using AI tags from Crunchbase<\/a> and Pitchbook<\/a><\/p>\n<\/li>\n

  2. \n

    Add general up-to-date information from the startup main page. We used a parsing script (Python) to get data and automatically translate it into English. This data was merged with data from Crunchbase<\/a> and Pitchbook<\/a>. It took 2 minutes.<\/p>\n<\/li>\n

  3. \n

    Apply an AI model in Python script to extract technologies (Entity Recognition) mentioned in the descriptions. We used the ensemble of three AI models: roberta-base-squad2<\/a> (extractive model), GPT 3.5 Turbo<\/a> (generative model) and microsoft\/phi-2<\/a> (generative model). You may look at some technical details of implementation in the Colab Notebook here<\/a>. Model Recall is over 92% in the AI dataset for technology extraction tasks. You will get the list of candidates, that is to be cleaned by experts to remove false positives (in some cases AI models confuse technology with products or generate «hallucinations»). You also have to merge synonyms into one technological entity. It takes around 1-2 hours in total.<\/p>\n<\/li>\n

  4. \n

    Prioritize technologies by the number of startups that mention this technology in the description (A) or by total investments (B). 1 minute with script and Pitchbook\/Crunchbase API.<\/p>\n<\/li>\n

  5. \n

    Prepare Top10 list of technological trends. Analyze the drivers, constraints and the prospects of each trend using an expert assessment. 1 day for analysis of reports and reviews through Perplexity<\/a>.<\/p>\n<\/li>\n

  6. \n

    Apply generative model, GenAI (for example GPT Turbo) to extract basic applications and Use Cases for each technology. Collection of startup descriptions is used as the input. 5 minutes.<\/p>\n<\/li>\n<\/ol>\n

    \"AI<\/p>\n
    AI — Expert Collaboration in Trendwatching Pipeline<\/figcaption><\/div>\n<\/figure>\n

    The total time for analysis is around 1-2 days. <\/p>\n

    The resulting list of Top 10 technologies looks like this: Generative AI<\/strong>, Computer Vision<\/strong>, Predictive Analytics<\/strong>, Data Insights and BI<\/strong>, AI Tools<\/strong>, AI in Cybersecurity<\/strong>, Semantic Search<\/strong>, AI in Blockchain<\/strong>, Transparent and Safe AI<\/strong>, AI in Edge Computing<\/strong>.<\/p>\n

    Let’s look at the example of Top 5, most popular Use Cases for Transparent and Safe AI: diagnosis of diseases and treatment recommendations (1), creditworthiness analysis (2), transactions monitoring (3), investment decisions (4), defense and security (5). Not bad for the MVP.<\/p>\n

    What is the value of this methodology? Clear logic, use of primary<\/u> data, high objectivity \u0441onsequently. You can use it on a regular basis (per quarter, per month), use academic papers and patents for monitoring emerging trends<\/u> as well. <\/p>\n

    Pipeline: Business Trends<\/strong><\/p>\n

    The current approach has one drawback: we miss business models and products that don’t mention technology. Actually sometimes innovation lies within the business model — Uber, BNPL players and so on. So, we need to change something in our model to move towards a more flexible approach. This time we look for trends in Q1 2024, FinTech industry:<\/p>\n

      \n
    1. \n

      Get startup descriptions (2024, Q1) using FinTech tags from Crunchbase<\/a> and Pitchbook<\/a><\/p>\n<\/li>\n

    2. \n

      Add general up-to-date information from the startup main page. We used parsing script (Python) to get data and automatically translate it into English. This data was merged with data from Crunchbase<\/a> and Pitchbook<\/a>.<\/p>\n<\/li>\n

    3. \n

      Let’s ask GPT Turbo API<\/a> «What are the most popular diversified trends without company titles (in the format ‘Technology or Product — Value for the user’) mentioned here. Give a list of 10 items». Collection of 121 startups splitted in chunks is the input. Context provided: «You are a FinTech expert». GPT Turbo parameters: temperature = 0, top_p = 0.9, presence_penalty = 2. <\/p>\n<\/li>\n<\/ol>\n

      The resulting list looks like this<\/u>:<\/p>\n