At the beginning of the year it looked like 2014 was shaping up to be a year when Small Data moved out of the shadows and into the mainstream (or maybe even THE Year of Small Data as I predicted in this op-ed for ZDNet). Judging by the volume of articles, social chatter, and casual conversation I’ve had at numerous industry events, Small Data has arrived. But it still means different things to different people. And that’s cool, since there’s a lot we’ve learned over the past couple years about why it’s important – not only as an alternative to Big Data – but also as a design philosophy and movement that shifts the conversation from processing power…to people!
So if you’re new to the topic, or just curious how Small Data has grown up, I’ve collected my “Top 10” articles for sampling the early concepts as well as the latest thinking. There are many more pieces I’ve left off, but this might serve as a core reading list. And yes, I’ve included just 2 of my own articles, but if you like them you can find many more on this blog or on the Actuate buzz page.
The Early Days – Foundations
Forget Big Data, Small Data is the Real Revolution – Rufus Pollock’s (@rufuspollock) original post on the power of Small Data, collaboration and “decentralized data wrangling” from April 2013. Consistently one of the top 5 most cited pieces on the topic, it’s a great companion to my Forbes piece below.
These Smart, Social Apps Bring Big Data Down to Size – my piece with Mark Fidelman (@markfidelman) from October 2012 that introduced the pillars of small data design: make it simple, be smart, think mobile (be responsive) and predicted a new wave of user-centric business intelligence.
What happens when each patient becomes their own “universe” of unique medical data? – Prof. Deborah Estrin (of @cornell_tech) talk at TEDMed 2013 has key ideas around managing our own personal Small Data. Especially relevant to not only privacy discussions but also those pondering the next wave of wearables and fitness tracking devices.
Data-driven Marketing – 3 Perspectives
Small Data Can Help Businesses Be More Human – great post on the need for “human scale” in marketing by Brand Networks CEO Jamie Tedford (@jamietedford).
What the “Small Data” Revolution Means for Marketers – it’s all about delivering more targeted, more personalized and all-around more intelligent marketing campaigns says CommandIQ (@CommandIQ) CEO Noah Jessop. Right on!
What’s The Big Deal About ‘Small Data’? – recent roundup of perspectives on the marketing front (yup, including my own) by my friends at CMO.com.
Is Small the New Big? – Mainstream Thinking and New Voices
Focus on data value, not its ‘bigness’ – one of my recent op-eds on data value, decentralization and the new CRM. Good overview piece if you want to get a high level perspective and stir up thinking within your own organization.
In Praise of ‘Small Data’: How Targeted Analytics — Not ‘Big Data’ — Are Transforming Education Today – new post by Brian Kibby (@BrianKibby), President of McGraw-Hill’s Higher Education Group.
Your Sushi May Be Getting Smarter – new piece in The Atlantic by Tech Editor Adrienne LaFrance (@AdrienneLaF) on “smart objects,” food safety and the power of Small Data (and IoT).
How To Create Incredible Customer Service Through The ‘Small Data’ Advantage – new piece by customer service speaker and author Micah Solomon (@micahsolomon). While a lot of my focus to-date has been on marketing use cases, I think there are some great ones in customer service/loyalty as well as this post suggests.
What articles did I miss? What would you add if I made it a “top 20” list instead?
In a thought provoking interview with CNET published this past week, Fitbit designer Gadi Amit explores the use of wearables in everyday applications – and introduces the notion of “wise” devices that provide just the right information, when and where we need it.
Beyond the fact that Amit’s firm is designing wearables for unique markets like babies (well I guess really for their parents) and pets (!), what struck me in this piece was Amit’s perspective – certainly shaped by his role as president and lead designer at design firm NewDealDesign – on the state of wearables, their future, and our relationship with them. Specifically, in response to a question about how wearables will be integrated into our daily lives, he states:
The interesting thing is when I say that, people immediately jump to the conclusion that we will be cyborgs. My goal with designing this is that we won’t be cyborgs. We actually will become more human and more free from the technology. What we have now in the design business is two camps: there is the camp that wants to create a lot of data and wants to analyse a lot of data; and there is the other camp which I belong to that tries to create devices that are not smart, they are actually wise. They are more than smart, they are wise enough to understand you, to filter and allow you to go on with your life with all their data processing in the background giving you hints of what is essential when it is essential.
Having data processing in the background and focusing on what information is essential is of course very much in line with the small data “aesthetic” we’ve been promoting here and in a number of venues over the past 2 years, so it’s cool to hear validation from another corner. As a former AI/machine learning guy, I also like the idea of “wise” devices that understand context and personal preferences, and can make a case that small data will in fact be the new “OS” for these devices (more in a future post).
But even more so, if we think of the cyborg comment as a challenge to all of us, I think we need to consider the element of “humanness” as we create new apps and digital experiences. And perhaps provide better opportunities and incentives to untether/unplug (partially?) from our digital devices, even as consumers clamor for faster, more personal, more portable, and ultimately more satisfying data-enriched experiences.
Designing Data-driven Apps
Speaking of the new data consumer, I’ve been spending more time with developers and those thinking about the future of customer facing apps, and recently created a talk on design principles that builds on some of the work you’ve read about on this very blog. As always I believe that data-driven design is an art and a science, so it’s been fun to brush up on the science/tech part for sure.
Of course our first job is still to think about the end-consumer, and how we can inform, connect, and motivate them to get involved or take action. As an aside, if you’ve paid attention to how I’ve presented this last point, I’ve always used Nike Fuelband as my example, so with news that Nike is getting out of the fitness hardware business (good analysis in this Gigaom piece), it’s been interesting to see Fitbit and even Samsung step up their efforts ahead of the likely fall iWatch debut.
On the business side, beyond understanding the value of data along the customer journey and focusing on “last mile” functionality, having a scalable foundation that can potentially support millions of users and large data sets from many sources (before it is transformed into useful small data) is essential as we look to bring powerful, yet human-scale, smart (wise) apps to the masses. So is a community to drive innovation – like the 3.5 million BIRT developers, or 600K+ Drupal users and coders.
Many of these ideas (and some examples) were covered in the talk I did with SD Times recently. There’s a link to the replay and a summary by my colleague Fred Sandsmark on the Actuate blog – which you can read here.
I also presented a longer version focused on bringing the power of advanced analytics to “everyday tasks” at the CAMP IT big data event this past week, (a well-produced event by the way) and plan to post those slides to my slideshare shortly.
Finally, I will be moderating a very cool expert panel on “building the next big app” at a special event Actuate is hosting in San Jose on the evening of July 10. Scheduled to join me on stage will be Eclipse Foundation Executive Director Mike Milinkovich, plus industry watcher and enterprise apps futurist Esteban Kolsky, along with 1-2 other special guests. We’ll explore how consumer experiences will (and are) be shaped by new devices and data, open source driven innovation, and next-generation design tools and practices.
Be sure to let me know if you’ll be in the area and want to join us, since I have a limited number of VIP passes to share.
The small data movement continues to gain followers. In fact, just in the past couple weeks we’ve seen articles in a wide range of publication including Ad Age, CITEWorld (smart data!), InformationWeek India, RetailWeek, and the SmartData Collective, as well as new opinion pieces on the problems with big data in the New York Times and FT magazine. Meanwhile one of the more popular sessions at Adobe’s marketing summit tackled the topic and featured my former colleague Scott Liewehr who reviewed findings from the study we did at DCG last summer (see nice recap of the session by Trueffect here).
As I’ve discussed over the past 2 years, many of the benefits of thinking small focus on making personalized insights more pervasive via new, smarter devices, tools, and apps. Bringing big data down to size helps both decision makers and “everyday” business users or consumers find (just the right) information/content they need, in a format that makes it super clear and easy to take action. This to me is the essence of what it means to be data-driven, and in many ways is powering the excitement around wearable technology like Pebble, Fitbit, and Google Glass.
So, for consumers, small data continues to be about connecting (with data and peers), convenience, and portability. Yet for businesses looking to tap the power of big data for everyday decision making, it’s also about speed. In fact, one could argue it’s mostly about speed and being faster to market and more responsive to customer needs. How so?
As brands strive to be more data driven in their product development, marketing campaigns, and experience design, thinking fast means getting actionable insights in minutes and hours vs. days and weeks as we’ve become used to in “traditional” BI and Big Data (thanks to having to wait for IT resources or our neighborhood data scientist to pay us a visit!).
In today’s business environment, it’s increasingly tough to justify these bottlenecks. Especially when many business users are already part of the way there with tools like Excel or Tableau, and others have seen the potential of a small data approach via their experiences with consumer apps and Websites.
The bottom line? When social and mobile are the rule, and data increasingly drives competitive advantage, speed, and being more responsive, is the new “big.” For business users, this starts with being ruthlessly focused.
Staying Focused to Move Fast
Moving fast, failing fast, even being an agile fast follower, are all proven business strategies. This is the case for organizations in growth industries, and especially for leaders who need to deal with external competitors as well as internal adversaries like budget cycles or organizational inertia. But to go fast, you need to have focus. And think small (and fast), when it comes to data access, discovery, and delivery.
In business, being focused is critical to bringing benefits of big and small data to the broadest set of users. And often requires that we reduce scope to just the essential questions/algorithms/visuals needed to help business analysts and their peers understand the essence of their data and optimize campaigns and experiences in near real-time.
Building on our small data design principles, this means striving for simplicity in our tools, being smart “enough” to handle role-specific tasks, being responsive in terms of portability AND delivering immediate value, and making it easy to compare notes and socialize findings.
Customer Analytics for the Rest of Us
Digging deeper, if we review our definition for small data, the last part (be accessible, understandable, and actionable, for everyday tasks) effectively becomes our checklist for designing and evaluating prospective customer analytics tools that are geared to the non-data scientist and everyday sales, marketing, and service tasks (like Actuate’s BIRT Analytics), as follows:
- Is it fast and simple to access all the data we need – from CRM, ERP, Web, campaign tools, social channels, and other sources – to create a true picture of customer and market needs (and easy to clean the data if needed)?
- Can non-technical users explore, enrich, and understand this multisource data in one place – creating an analytics “sand box” as Boris Evelson from Forrester has called it?
- It is easy for business analysts to create visualizations, share findings, and apply insights in new campaigns and experiences (via workflow and connectors) without needing expert assistance?
- Are common “everyday” tasks/use cases like segmentation, cross-sell campaigns, or social/Web targeting supported – and are there pre-build algorithms tied to these use cases?
Moving fast shortens the time to value, and allows more time for socializing insights and field-testing new campaigns and offers (as a type of “rapid prototyping” perhaps). Doing so requires simpler and smarter tools, and removing IT bottlenecks. But most importantly it means putting the power of advanced analytics in the hands of everyday business users.
I started programming in 1980. In 1990 I finished my master’s degree focusing on machine learning. In 2000 I spent a year at McKinsey developing economic models for online advertising. In 2010 I co-founded Offerpop, which is now the leading social marketing platform for Facebook and Twitter.
You could say I have been around data, and data processing, and data analytics all my life! But I’ve never seen the changes – and opportunities – I’m seeing today.
With 1 billion smartphones shipping each year, the potential of big data analytics, the Internet of Things ramping up, and wearable devices like Glass and Nike Fuelband in the public eye, we have in front of us seemingly limitless options for generating, sharing, and displaying our data. Meanwhile, as my former colleague Dr. Tim Walters has artfully described: “Ubiquity negates scarcity.” In other words, the connected consumer (not their bank, or favorite retailer, or media outlet) calls the shots in this post-digital world. And those brands and service providers who are most responsive, and engaging, and useful to the connected consumer are going to clean up.
How? It’s all about data of course. Or more specifically about providing compelling data-enriched experiences that bring the power of big data to everyday tasks. These are the charting tools that E*TRADE provides that make an amateur investor feel like they can compete with the pros. Or Kayak’s awesome “When to Book” tool that lets you know if it’s the best time to purchase your airline ticket.
They are apps that present just the right information, on the channel of choice, at the right time. And even seem like “magic” because they anticipate your needs or make it super-easy to discover or share new insights. They are simple, smart, responsive, and social too – following the core principles of the small data design philosophy.
While there are already some excellent early examples, turning this theory into (broad) practice and fully understanding and mapping out the blueprint for tomorrow’s customer facing applications provides a compelling (and potentially lucrative) challenge – and requires the right partner.
A new challenge, a new role
Today I’m excited to announce that I’ve joined Actuate as their new VP of product marketing & innovation to move forward many of the ideas I’ve shared here on this blog and tackle the challenge above (with a lot of helpers!). Why Actuate? To be able to work with the team who literally invented enterprise reporting and reinvented it with BIRT, and have access to a super-smart technical group and community of 3M developers offers a “test bed” for some of my ideas I could only dream of. Plus there’s a great story forming around Actuate’s 3 new business units, an executive team I’ve known and trust (for almost 20 years), and a marketing team with the resources in place to build something special.
What’s on my plate for the first few months? In addition to exploring ways to drive market awareness for the company’s BIRT Analytics offering, I’ll be leading the design of a new company-wide content marketing “hub” concept that leverages my work in this area over the past few years, and will be working my industry contacts for new research partnerships and growth opportunities as well.
Oh, I’ll still be writing, speaking, and spreading the word about the power of small data and personalized analytics. And I also plan to expand the content (more posts!) and contributor roster right here to leverage Small Data Group’s position as the top ranked site for “small data” on Google.
As always I truly appreciate everyone’s feedback, encouragement, and support!
Big data has never been bigger. On the heals of a new study (reported here) that shows that investors have pumped $3.6 billion into big data startups this year, and Gartner’s latest Hype Cycle which shows big data quickly approaching the top of the hype “cliff” (along with Consumer 3D Printing and Gamification), one is tempted to see more of the same in 2014. Yet all the steam coming out of the big data hype machine seems to be obscuring our view of the big picture: in many cases big data is overkill. And in most cases big data is useful only if we can do something with it in our everyday jobs.
Which is why against the backdrop of more hype (and more confusion) about the real role for big data, the case for thinking small looks more and more attractive. In fact, I recently penned an opinion piece for ZDNet in Paul Greenberg’s excellent column on Social CRM on 10 Reasons 2014 will be the Year of Small Data. You can read the full piece here, and here’s the Cliffs Notes version of the first 5 reasons:
- Big data is hard. Doing it at scale takes time, especially when the tech guys own it (I used to be a data scientist, so I can say this!).
- Small data is all around us. Social and mobile signals can help us understand customer needs – today.
- Small data powers the new CRM. Small data is central to the rich profiles essential to targeting offers and experiences.
- ROI is the thing. To realize the full value of data-driven apps they must be accessible, understandable, and actionable for everyday work (remember our definition).
- Data-driven marketing is the next wave. As more consumers want to access, and consume and even wear useful data, there’s an unprecedented opportunity for savvy marketers and platform providers like Adobe, HubSpot, and Salesforce, along with data specialists and tool providers like Actuate, Localytics, TrueLens, and Visible to power the next wave of apps, experiences and devices.
What do you think? Are you finalizing your big and small data strategy for 2014? If you’d like DCG’s perspectives as you formulate your plans I’ll be hosting a special Webinar on Dec 12 at 1:00pm ET to share our outlook and 2014 research agenda. Hope to see you there!
UPDATE: missed my Webinar? You can view the replay by clicking here.
One of the best parts of my research over the past several months as I worked on DCG’s “Bringing the Power of Big Data to the Masses” study (report now available here) has been getting to sample both current viewpoints as well as some of the literature and developments from the past 20+ years on the topic. In fact, big data and today’s analytical tools are very much a product of the “generations” of data and data processing that came before them.
Here’s a cool illustration we created for the report that highlights some of the key developments, deals, and publications related to big data, analytics, and small data since the invention of the web in 1989. We had to leave out a number (many) of potential items, so I’d love to hear what milestones would you have included – enjoy!
With a flurry of new articles on small data in various publications recently – including Wired, Technorati, and AllThingsD – there’s a growing number of voices contributing to the small data movement (a great thing!). But with these new perspectives, I feel like it’s a good time to re-ground the conversation in an actual definition of what we are talking about when we refer to “small data” – or at least what I’m talking about!
So in this post I wanted to share my first cut at a proper definition. Yes, I’ve framed the pillars for small data in many other places, going back to my first guest piece in Forbes on the topic a year ago (!), and more recently I’ve embraced the idea of describing small data as “the last mile of big data.” But these were descriptions or principles vs. definitions for the most part.
So, after spending the last couple month working on Digital Clarity Group‘s new multi-client study on “Bringing the Power of Big Data to the Masses” (sponsored by my friends at Adobe, Actuate, HubSpot, and Visible), seeing how marketers are looking to make analytics more accessible and actionable – and creating some new starter use cases, an interesting thing happened: a definition emerged!
In fact, while our final report is still a week or so away from being available from DCG and our sponsors, I shared the definition with our audience at the Digital Pulse Summit this past week during my panel on small data, and given the response, I wanted to provide it here as well.
A New Definition for Small Data
Small data connects people with timely, meaningful insights (derived from big data and/or “local” sources), organized and packaged – often visually – to be accessible, understandable, and actionable for everyday tasks.
Note, as we describe in our report, this definition applies to the data we have, as well as the end-user apps and analyst workbenches for turning big data set into actionable small data. The key “action” words here are connect, organize, and package, and the “value” (the 4th V of big data) is rooted in making insights available to all (accessible), easy to apply (understandable), and focused on the task at hand (actionable).
In fact, I hope it’s as much a mission statement, as it is a definition. What do you think? Did we nail it?
Reaching and engaging today’s social, mobile consumer creates unique challenges for retailers. With more than one billion Facebook users, and nearly everyone on the planet with a mobile phone, there’s never been more potential to reach new audiences, deliver compelling omni-channel experiences, and gather new insights on consumer likes and dislikes. Yet these same trends have also created new distractions, potential threats (and opportunities) from the phenomenon of “showrooming,” and a shortage of expertise for sifting through and make sense of a growing mountain of customer data.
Against this backdrop, the importance of connecting with buyers via tailored offers, recommendations, and experiences has never been more important. Even in B2B, among best-in-class content marketers, 71% tailor content to a profile of the decision maker, according to a recent CMI/MarketingProfs study. But at the same time, delivering convenience and bringing your brand and data to where your customers are, and factoring in the role of influencers, product reviews, and word of mouth needs to be in the mix as well. Clearly, in this environment, merchandising is both an art and a science!
This is why I’ve been seeing a tremendous opportunity for retailers who embrace their customer data as a both an asset and a product, and also invest in the notion of small data by tapping digital breadcrumbs, social signals, and profile information (as sources) and committing to making insights actionable and available to (the broadest set of) staff and customers alike. In fact it’s a theme that I’ve validated in several discussions with retail consultants, planners, and merchandising professionals over the past several months.
The end goal is greater understanding, better performing campaigns, and a more rewarding experience for your customers.
More specifically, as I recently discussed in a Digital Clarity Group Brief sponsored by my friends at SDL (you can download the paper as well as some excellent presentations here), it’s clear that:
- Retailers can boost their understanding of customers and better serve them (on their own terms) across all channels by looking to blend transactional processes and insights with social interactions and data.
- Innovators are focusing on making insights actionable via recommendations and predictive targeting, smart apps/kiosks for associates, and location-based offers.
- There are clear benefits to thinking small – targeting local campaigns and data, starting with high potential segments, and streamlining key stages of the path to purchase.
Most importantly, we can’t forget to ask customers what they think. Encourage social feedback and make it easy to share offers with friends. If you can’t measure it (conversions, sharing, etc.), don’t do it!
The bottom line as retailers look to position their products, create timely and compelling offers, and deliver on the promise of true omni-channel commerce, is to leverage all insights to streamline back-end processes and simplify front-end interactions. This means chunking down the path to purchase, and looking for ways to present the right product at the right time, offer help (if needed), and make it easy for consumers to finish their journey.
What do you think? Do you buy it?
Last week I gave the closing keynote at IMS San Francisco on the topic of Rich Media, Personalization and Small Data. The event – which brings together top marketers and thought leaders in content marketing, social media and advertising – was a great forum for testing some of my latest thinking on content and scale. And specifically the growing shift from big budget campaigns, long-form video, and big data, to word of mouth viral campaigns, short-form content like vine and snapchat, and small data.
As I shared, this shift impacts (and enables) how marketers can create more effective way to reach and engage customers AND employees, and how they can bring their stories to life. To do so, I argued we need to align around 3 goals:
- Inform – Be HELPFUL! People pay attention if you’re saying something useful or unique – and you reach them when they are thinking about what you are saying,
- Connect – Help customers connect with you and each other…and
- Motivate – The essence of influence is motivating a behavior; in the digital world, content + context drives participation. The ultimate goal of content marketing (and small data) is creating this action or behavior. No participation or action = NO ROI!
With these goals in mind we can map out specific tactics like making video content and your YouTube channel part of your product strategy, and bringing data to where your users are, via mobile apps, social coupons or QR codes.
You can check out my entire presentation and see other examples and takeaways in my SlideShare below. Enjoy!
As I prepared my presentation on small data for the recent DCG Insights day in New York, I did a quick review of some of the latest coverage of the topic, and discovered a number of new articles worth checking out. This list just scratches the surface, so I’ll add to it in subsequent posts, but wanted to share this first batch as a reading list for those interested in the topic. Enjoy!
5. Inc.com – Why Small Data May Be Bigger Than Big Data – in this piece which came out in April, x-McKinsey consultant and loyalty expert Victor Ho argues that small data is the key to solving the “data divide” for local businesses.
4. ITworld – Big Data Benefits with Small Data – tech journalist Brian Proffitt questions the necessity of big data for non government agencies or businesses that don’t have giga-scale ecommerce sites, and references the notion of personal data stores – a distinct small data use case. His closing point is a good one:
“Thanks to big data, many businesses recognize the value of data analysis. But there may be several new paths that will open up to help them achive the benefits of data decision making.”
3. MediaPost – Does Big Data Require a Big Rethink? – in this commentary Michael Hemsey of Kobie Marketing urges brands to focus on “the little details,” calls out his recommendations for thinking small when it comes to data assets, and talks about the age of mini-measurement – which is a cool way to put it!
“I don’t want to have a relationship with a marketing department. I don’t want to be your friend. I don’t want to engage in conversation with you. I feel no loyalty towards you. When I say I like you I’m not entirely sincere.
And yet I chose to share an enormous amount of my life with you. The detail I’m able and happy to share has grown big. Really, really big. But understand this: my reason for sharing this data is entirely motivated by self-interest. You see, I know as much about you as you do about me. I know how valuable my data can be to you. So I expect you to use this data for my benefit. Because you can be damn sure I will be.”
His conclusion is that small data is specific and concrete, which makes it easier to to “make good use of it.” And more so, requires us to understand our customers, their lives and where we fit within them. This is exactly the connection I’m looking to make between the worlds of CRM and small data, in the research I’m ramping up for Digital Clarity Group.
1. The Guardian (UK) – Forget Big Data, Small Data is the Real Revolution – one of the most widely referenced pieces of the last few months, Open Knowledge Foundation founder Rufus Pollock frames big data as the latest “centralization fad” and notes:
“the real opportunity is not big data, but small data. Not centralized “big iron”, but decentralized data wrangling. Not “one ring to rule them all” but “small pieces loosely joined.”
I love the point about decentralization, which ties nicely into the social aspect of small data. In fact Pollock ends with a great framing of small data as part of a larger movement driven by digital disruption and democratization of IT:
“This next decade belongs to distributed models not centralized ones, to collaboration not control, and to small data not big data.”
In terms of older articles, check out my guest post in Forbes from last October, and the early piece by Patrick Gray in TechRepublic that argues for a practical, consolidated approach to data and reporting.
What articles did I miss? What’s on your “must read” list?