Future of Engagement #8: Collective Intelligence

Organizations synthesize search, social and sensor data streams into insights that guide smarter actions.

What is Collective Intelligence?

Collective intelligence involves analyzing the collective actions and feedback of people, finding patterns and trends, and sharing it back to aid understanding and guide action. Organizations, artists and changemakers are using collective intelligence to analyze opinions and behaviors, identify patterns and trends, and recommend actions or inspire change.

The rise of collective intelligence can be attributed to three broad trends. First, people are sharing immense amounts of location-based, personalized data online, both implicitly by searching, clicking or buying and explicitly by creating profiles, posting status updates, and checking in to locations and events. Second, people are beginning to use sensor-based devices to track and share real world data about our bodies (quantified self) and our devices, houses, and environments (internet of things). Third, web platforms like Google, Facebook, Twitter and LinkedIn are anonymizing and aggregating this data, mining collective intelligence from it themselves, and also making it available for third-party applications via robust APIs.

Web platforms are using data to create reviews of the most important trends and events in the previous year (Google Zeitgeist(video), 2012 Year on Twitter (video), Facebook Year in Review); add new perspective to important political, sports and entertainment events (Amazon Election Heat Map (screenshot), Twitter Political IndexFacebook America Votes 2012 (video),Twitter Oscars Index); and even predict potential career paths (LinkedIn Career Explorer (video)), the spread of communicable diseases (Google Flu Trends (video)) and traffic conditions (Google Maps Traffic (video)).

News and entertainment media organizations are partnering with internet platforms or using their APIs to use search and social data to analyze public opinions, predict the outcome of important events (USA Today/ Twitter Election Meter, Facebook/ CNN Election InsightsE! Heat Gauge (video)) or showcase upcoming artists (MTV Music Meter (video)).

Several web platforms are finding patterns in user profiles, networks and behaviors to make better product, movie, book, music and restaurant recommendations (AmazonNetflixRandom House’s BookscoutGoodreadsPandoraBundle).

Entrepreneurs and changemakers are creating niche platforms to mine social and search data to improve traffic conditions (Waze (video)), optimize energy consumption (Opower (video)), and aggregate health data to predict outbreak of diseases (Sickweather (video), Flu Near You (video), HealthMap (video)) and even explore effective cures (Patients Like Me (video), NextBio (video)).

Next Bio

Some collective intelligence initiatives have achieved significant impact and scale. For instance, Waze’s community of 36 million drivers shared 90 million user reports on real time traffic, accidents, hazards, police, gas prices and map issues, and Opower has used data from 80 utility companies to help reduce energy consumption by 2 billion kilowatt hours and save $234 million on electricity bills.

The success of such collective intelligence platforms shows that it’s possible to synthesize search, social, sensor and self-reported data from millions of people into meaningful real-time insights that can guide actions and change behaviors at scale.

How does Collective Intelligence work?

Collective intelligence platforms can be classified across three dimensions: the type of data, the method of data analysis, and the possibilities for participation.

Most collective intelligence platforms use a combination of search, social, sensor and self-reported data. Recommendation engines (Amazon) primarily use on-site browsing, buying and rating data, but are beginning to add social data. Navigation apps (Waze) primarily use automatically updated location data from smartphones, with some self-reported data. Many behavior change applications (Opower) use sensor or transaction data from their own or partner devices, but sometimes add in social data. Many platforms from media and entertainment organizations (MTV Music Meter) use social data sourced from social network APIs. Platforms that use search, social or sensor data typically use the public APIs or take a one-time permission from the user. Platforms that use self-reported data from specialized communities often build their own community platforms and add gamification features to encourage people to share data regularly (Patients Like Me).

Different collective intelligence platforms synthesize data in different ways. Some platforms use algorithms to cluster users and products based on viewing, buying, or rating behaviors and show their recommendations in terms of “users who liked these products also bought these other products” (Amazon) or “users who have similar characteristics also behaved in this way” (Opower). Many platforms plot search, social, sensor and self-reported data on maps, based on keywords or metadata, to find shifts in geographical patterns over time (Sickweather). Other platforms find patterns in social conversations through text and link analysis and connect them back to source or profile data (Facebook/ CNN Election Insights). Some platforms allow users to filter through the data based on time, location, popularity or sentiment to get to more nuanced insights.

Many collective intelligence platforms have overlaps with co-creation communitiessocial curation platforms, and behavior change games, and offer similar possibilities for participation. Crowdsourcing-driven platforms ask users to create profiles, share answers or ideas, and engage with other users’ content (Patients Like Me). Curation-driven platforms ask users to engage with other users’ content and tag their own content so that it might be included (Sickweather). Behavior change driven platforms compare the users’ behaviors with similar others and incentivize them to change their behavior through gamification features (OPower).


Collective Intelligence for Brands

Many organizations and brands are experimenting with collective intelligence in meaningful ways.

A number of organizations have created ideation platforms to crowdsource insights from employees, partners and customers, and some have even used these insights to create new product and service offerings (Dell Ideastorm, MyStarbucksIdea). Many other organizations have created long-term public or private insights communities to get a more nuanced understanding of consumer behavior, and some have even shared these insights back with the community. For instance, Nestle launched the Happily Healthy Project (video) quiz to help Australians measure their Happily Healthy Quotient and compare it to nation and state averages, filtered by a number of demographic variables like age, gender and income. Other organizations have partnered with independent community platforms to get insights about specialized high-value communities. For instance, several pharmaceutical companies have partnered with Patients Like Me to understand patient needs and drug performance.

Other organizations have taken the social curation route to synthesize and share insights from social conversations around important events. For instance, KPMG built WEF Live to curate the conversations around World Economic Forum and highlight the most important themes from WEF delegates and WEF watchers from around the world. During the 2012 London Olympics, GE tied up with NBC to track Twitter conversations around the games. Almost all major brands are trying to use big data, including search and social data, to understand and engage with consumers. For instance, Vicks combined aggregated search data from Google Flu Trends with demographic data to target moms in high flu zones with ads for their premium Flu Thermometer. @WalmartLabs has analyzed vast amounts of social data (“fast data”) and combined it with public web data and proprietary data to create the social genome, a living database of entities (people, events, topics, products, locations, organizations) and their relationships. It is now building a series of collective intelligence social applications using the social genome, starting with the social gift recommendation app Shopycat (video).

Finally, some organizations are building platforms and products to synthesize and share insights from sensor data, both in the quantified self-space and the internet of things space. For instance, the Nike+ (video) and Adidas MiCoach (video) range of wearable sensor-enabled products enable people to track their workouts, compare themselves with friends and similar others, and even compete with others. Audi partnered with MIT to create a Road Frustration Index (video) based on traffic and weather conditions, reported accidents and driver sentiment from social data.


Future of Collective Intelligence

In the near future, we expect more social platforms like Google, Facebook, Twitter and LinkedIn to synthesize user data to share insights that help users get a new understanding of their own behaviors (how families interact on Facebook). We also expect social platforms to create more data-driven applications that help users make meaningful decisions and change their behaviors (LinkedIn Career ExplorerGoogle Flu Trends).

We also expect the social data space to explode with new, specialized players. GnipTopsy and DataSift (video) aggregate data from multiple social platforms, provide applications to recombine and analyze them, and APIs for third party developers to build applications on them. Other data players are focusing on building social data applications for a specific industry: Dataminr for financial services, BlueFinLabs (video) and Second Sync (video) for Television, Next Big Sound video and The Echo Nest for music, and ReviewPro for hotels. We also expect other data startups to focus on sensor data (SensorCloud (video)) and transaction data (Swipely (video)).

We expect that big corporations will acquire many of these social data startups. For instance, Twitter acquired TV social data startup BlueFin, Intuit acquired personal finance startup Mint, eBay acquired personal recommendation startup Hunch, and Walmart acquired social commerce startup Kosmix (now @WalmartLabs). Other organizations will partner with platforms like Kaggle or DataKind to run crowdsourced data challenges.

We also expect that organizations will shift their focus from collecting and analyzing data to creating applications that use the data to help their users get better understanding and make better decisions. As a result, social curation tools like MassRelevance, insight community tools like CommuniSpace and crowdsourcing tools like BrightIdea will all strengthen their features around visualizing and showcasing data back to the users to guide action.

Finally, we expect that “fast data” will be the next big thing after “big data”, as organizations seek to analyze data streams from social conversations, search queries, sensors, and transactions, find patterns and actionable insights, and share it back with users to help them make better decisions, all in real time.

- X – X – X -

This is a chapter from a MSLGROUP report titled “Now & Next: Future of Engagement” that I wrote with Pascal Beucler and Nidhi Makhija.

The report highlights the ten most important frontiers that will define the future of engagement for marketers, entrepreneurs and changemakers: CrowdfundingBehavior Change GamesCollaborative Social InnovationGrassroots Change MovementsCo-creation CommunitiesSocial CurationTransmedia Storytelling, Collective Intelligence, Social Live Experiences and Collaborative Consumption.

In each of these reports, we start by describing why they are important, how they work, and how brands might benefit from them; we then examine web platforms and brand programs that point to the future (that is already here); then finish by identifying some of the most important features of that future, with our recommendations on how to benefit from them.

Future of Engagement report on SlideShare

Future of Engagement slide deck on SlideShare

Future of Engagement video infographic on YouTube

Future of Engagement infographic on Visually