Just as embedded processors (and cheap memory) sparked a new generation of smart systems and consumer devices like the smartphone, embedded BI and analytics – enabled by advances in big and small data processing, open source projects like BIRT and Hadoop, and rich APIs – has the potential to change the face of many categories of applications (and devices).
Think hyper personalized and portable customer experiences, or smarter trading grids that anticipate disruptions or automatically seek out the best deal. Or new views into markets or business operations that reveal previously unseen relationships or potential innovations.
Meanwhile we are all looking to get closer to our customers, by gaining a true 360 degree view of what they want and how they are interacting with us and each other. This is where some of the new Analytics-as-a-Service (“AaaS”) offerings like Watson Analytics and the recently launched OpenText Big Data Analytics in the Cloud fits in. Combining advanced and predictive analytics, delivered as an easy to use managed cloud offering, AaaS aims to bring the power of big data to everyday business users, creating one view of their customer base, with a super-fast dedicated analytics data base and pre-built algorithms for handling the most common marketing and operational analyses.
The future of analytics is clearly about these types of tools that serve the growing population of “citizen data scientists.” It’s also about delivering insights from new data sources (think IoT) to users on their device of choice like smartphones, tablets or even smart watches. And building on a foundation of good information design (as detailed by Edward Tufte), blending the right unstructured and structured content, and applying the latest digital engagement models and approaches (like those from my friend and former colleague Esteban Kolsky).
Discover more in Vegas at Enterprise World
All of these scenarios will be front and center in the Analytics track at the upcoming Enterprise World 2015 event in Las Vegas in early November. I’ll be there with a cast of 1000’s delivering a couple keynote addresses as well as parts of several breakouts and demos looking at the role of big and small data, analytics and digital asset management, and helping to premiere our latest IoT demo (hint: it includes a network of Raspberry Pi-based sensors, an MQTT broker, and visuals powered by the OpenText iHub).
Learn more about the event and our sessions by clicking here.
If the desire to make analytics more accessible and actionable for everyday tasks helped kickstart the small data movement, the growing focus on the best ways to access/manage/visualize IoT devices and data offers to shift the small data discussion into hyperdrive. In fact, over the past couple months we’ve seen some great posts and discussions related to why small data is essential to harnessing the power of IoT. Here’s a review of 5 of them, including my latest article on the potential of new types of embedded intelligence. Enjoy!
1. Forbes.com – Forget Big Data – Small Data Is Driving The Internet Of Things – in this compelling, concise piece by Mike Kavis, he talks about the value of a hybrid small (and big) data approach when dealing with sensor data. His perspective:
“Small data knows what a tracked object is doing. If you want to understand why the object is doing that, then big data is what you seek.”
2. Economic Times (India) – EMC bets on ‘small data’ for future growth – interesting interview with EMC exec Guy Churchward, who argues that billions of interconnected devices like driverless cars will create “small data sprawl” – requiring new tools and approaches unavailable today.
3. Enterprise Apps Today – How IoT Will Change Big Data Analytics – in this roundup article there’s a number of scenarios, including a pretty cool section with an AT&T executive who points out the importance of information management:
“…determining what type of data is important, what should be transmitted immediately, what should be stored and for how long, and what information should be discarded. Otherwise, you could end up with an almost infinite pile of data to analyze, when only a relatively small portion is of real importance.”
4. Logistics Viewpoints – The Benefits of “Small Data” in Logistics – it’s always good to sample specialty industry perspectives, and this one features some good insights from ARC Advisory expert Ralph Rio, who talks about a “small data within Big Data” approach.
5. CTOVision.com – Beyond the IoT Buzz Is A New Horizon of Embedded Intelligence, Information Flows and Seriously Smart Apps – my latest op-ed for CTOvision, where I look at requirements for becoming data driven and some of our learnings from working with device data, including the role of new wearable computers as sources AND destinations for small data:
“…consumer wearables and other mobile devices have a dual role in this scenario as they not only create data, but they also consume (and display) it.”
What other articles or perspectives did I miss? I’d love to hear your ideas!
And if you are in London, Munich, Paris, New York or Toronto over the next few weeks and want to see our latest IoT demos in action, my team and I will be presenting at the OpenText Innovation Tour 2015 events – if you are interested in attending, check out the schedule here.
Back in the summer I explored the concept of “wise” devices being proposed by Fitbit designer Gadi Amit and introduced the idea that small data will be the OS for these mobile and wearable devices.
Since then, my teams at Actuate have been exploring these ideas and accelerating our work around applying information design best practices for a new generation of rich mobile apps and embedded analytics. The goal: expand the boundaries and our understanding of what it means to assemble and display intelligence in context. We’ve also been teaming with our engineering group to look at new ways to demonstrate the rich APIs provided by the BIRT iHub to better access real-time device data, visualize it, and embed these packaged insights on “non-traditional” devices like smartwatches, tablets or even large-format displays.
Some initial results – including a very cool IoT-telematics demo that leverages the BIRT technology stack and open data standards to show how to connect your car to your smartwatch – were first premiered by Actuate’s Kris Clark at EclipseCon Europe. And more recently this demo was updated and featured at our ongoing Data Driven Summit events, along with the IoS tablet app shown above which demonstrates rich, fluid visualizations in a mobile BI scenario.
These concepts and Actuate’s overall strategy for the Embedded Analytics market were also part of a media and analyst roundtable during our London event. And judging by the press coverage (see here, here and here), “Wearable BI” is a pretty compelling idea!
Observations from the Field
The great thing about getting into the field and sharing perspectives with practitioners and project owners around the globe (I’ve been in 6 cities on 3 continents over the past 3 weeks, speaking to and sharing with hundreds of attendees as part of our Summit program) is that we have fresh insight into how organizations are looking at visualizing their data, and where they are focusing in 2015 and beyond. As expected, mobile devices and data are a big driver, but so is the idea of operationalizing transactional big data by embedding analytics in more places, and making these insights more consumable by more users by raising the bar in terms of information design.
In fact my colleague Mark Gamble’s session on Visualization and UX Best Practices – featuring the best tips from Edward Tufte and Stephen Few and our own team – has been consistently one of the most popular and highest rated sessions during our recent tour.
Other notes, takeaways, and resources:
- The Small data philosophy is even more relevant today than when I started writing about it 2 years ago in Forbes. More analysts and companies are focusing on the “last mile of Big Data” as the place where value is created, especially when tackling everyday marketing challenges and designing data-driven apps for mobile and IoT. Since my last post some good articles to check out on the topic include these new pieces in Fortune and Forbes , my latest op-ed in CMSWire, and this cool infographic from Constant Contact.
- For data-driven mobile apps, a native/hybrid approach works best. Sure, native is great for games, but if you are creating real business apps, there are benefits of having some native code local to the device, and some non-platform-specific components resident on a back-end server to provide maximum flexibility and performance (and re-use!) for serving up rich visualizations that scale up to millions of users, and scale down to match individual preferences and limitations of small displays. For more detail, I’ve posted the slides from our Mobile and IoT session at Data Driven Summit on my Slideshare.
- Many organizations are looking to bring the power of big data to the masses. In fact this is arguably one of the top priorities for 2015. This means taking a fresh look at ways to turn data into information, and information into embedded intelligence, one of the themes Actuate’s CEO touched on at our Summits. It also means envisioning new use cases and new UXs for mobile (like our smartwatch app) – something BI guru Howard Dresner has proposed, and serving up alerts and contextual visuals that are foundational to the next generation of wearable BI and intelligent apps.
So, what’s on your list of project for 2015? I’d love to connect and hear about your plans.
Also, if you are in NYC on December 3 and want to see our IoT demo live and hear from customers, BIRT experts, and industry gurus about the latest thinking around data driven apps, there are still a limited number of seats for Data Driven Summit – New York. You can get more info and sign up (it’s free) here.
The small data movement continues to gain followers. In fact, just in the past couple weeks we’ve seen articles in a wide range of publication including Ad Age, CITEWorld (smart data!), InformationWeek India, RetailWeek, and the SmartData Collective, as well as new opinion pieces on the problems with big data in the New York Times and FT magazine. Meanwhile one of the more popular sessions at Adobe’s marketing summit tackled the topic and featured my former colleague Scott Liewehr who reviewed findings from the study we did at DCG last summer (see nice recap of the session by Trueffect here).
As I’ve discussed over the past 2 years, many of the benefits of thinking small focus on making personalized insights more pervasive via new, smarter devices, tools, and apps. Bringing big data down to size helps both decision makers and “everyday” business users or consumers find (just the right) information/content they need, in a format that makes it super clear and easy to take action. This to me is the essence of what it means to be data-driven, and in many ways is powering the excitement around wearable technology like Pebble, Fitbit, and Google Glass.
So, for consumers, small data continues to be about connecting (with data and peers), convenience, and portability. Yet for businesses looking to tap the power of big data for everyday decision making, it’s also about speed. In fact, one could argue it’s mostly about speed and being faster to market and more responsive to customer needs. How so?
As brands strive to be more data driven in their product development, marketing campaigns, and experience design, thinking fast means getting actionable insights in minutes and hours vs. days and weeks as we’ve become used to in “traditional” BI and Big Data (thanks to having to wait for IT resources or our neighborhood data scientist to pay us a visit!).
In today’s business environment, it’s increasingly tough to justify these bottlenecks. Especially when many business users are already part of the way there with tools like Excel or Tableau, and others have seen the potential of a small data approach via their experiences with consumer apps and Websites.
The bottom line? When social and mobile are the rule, and data increasingly drives competitive advantage, speed, and being more responsive, is the new “big.” For business users, this starts with being ruthlessly focused.
Staying Focused to Move Fast
Moving fast, failing fast, even being an agile fast follower, are all proven business strategies. This is the case for organizations in growth industries, and especially for leaders who need to deal with external competitors as well as internal adversaries like budget cycles or organizational inertia. But to go fast, you need to have focus. And think small (and fast), when it comes to data access, discovery, and delivery.
In business, being focused is critical to bringing benefits of big and small data to the broadest set of users. And often requires that we reduce scope to just the essential questions/algorithms/visuals needed to help business analysts and their peers understand the essence of their data and optimize campaigns and experiences in near real-time.
Building on our small data design principles, this means striving for simplicity in our tools, being smart “enough” to handle role-specific tasks, being responsive in terms of portability AND delivering immediate value, and making it easy to compare notes and socialize findings.
Customer Analytics for the Rest of Us
Digging deeper, if we review our definition for small data, the last part (be accessible, understandable, and actionable, for everyday tasks) effectively becomes our checklist for designing and evaluating prospective customer analytics tools that are geared to the non-data scientist and everyday sales, marketing, and service tasks (like Actuate’s BIRT Analytics), as follows:
- Is it fast and simple to access all the data we need – from CRM, ERP, Web, campaign tools, social channels, and other sources – to create a true picture of customer and market needs (and easy to clean the data if needed)?
- Can non-technical users explore, enrich, and understand this multisource data in one place – creating an analytics “sand box” as Boris Evelson from Forrester has called it?
- It is easy for business analysts to create visualizations, share findings, and apply insights in new campaigns and experiences (via workflow and connectors) without needing expert assistance?
- Are common “everyday” tasks/use cases like segmentation, cross-sell campaigns, or social/Web targeting supported – and are there pre-build algorithms tied to these use cases?
Moving fast shortens the time to value, and allows more time for socializing insights and field-testing new campaigns and offers (as a type of “rapid prototyping” perhaps). Doing so requires simpler and smarter tools, and removing IT bottlenecks. But most importantly it means putting the power of advanced analytics in the hands of everyday business users.
I started programming in 1980. In 1990 I finished my master’s degree focusing on machine learning. In 2000 I spent a year at McKinsey developing economic models for online advertising. In 2010 I co-founded Offerpop, which is now the leading social marketing platform for Facebook and Twitter.
You could say I have been around data, and data processing, and data analytics all my life! But I’ve never seen the changes – and opportunities – I’m seeing today.
With 1 billion smartphones shipping each year, the potential of big data analytics, the Internet of Things ramping up, and wearable devices like Glass and Nike Fuelband in the public eye, we have in front of us seemingly limitless options for generating, sharing, and displaying our data. Meanwhile, as my former colleague Dr. Tim Walters has artfully described: “Ubiquity negates scarcity.” In other words, the connected consumer (not their bank, or favorite retailer, or media outlet) calls the shots in this post-digital world. And those brands and service providers who are most responsive, and engaging, and useful to the connected consumer are going to clean up.
How? It’s all about data of course. Or more specifically about providing compelling data-enriched experiences that bring the power of big data to everyday tasks. These are the charting tools that E*TRADE provides that make an amateur investor feel like they can compete with the pros. Or Kayak’s awesome “When to Book” tool that lets you know if it’s the best time to purchase your airline ticket.
They are apps that present just the right information, on the channel of choice, at the right time. And even seem like “magic” because they anticipate your needs or make it super-easy to discover or share new insights. They are simple, smart, responsive, and social too – following the core principles of the small data design philosophy.
While there are already some excellent early examples, turning this theory into (broad) practice and fully understanding and mapping out the blueprint for tomorrow’s customer facing applications provides a compelling (and potentially lucrative) challenge – and requires the right partner.
A new challenge, a new role
Today I’m excited to announce that I’ve joined Actuate as their new VP of product marketing & innovation to move forward many of the ideas I’ve shared here on this blog and tackle the challenge above (with a lot of helpers!). Why Actuate? To be able to work with the team who literally invented enterprise reporting and reinvented it with BIRT, and have access to a super-smart technical group and community of 3M developers offers a “test bed” for some of my ideas I could only dream of. Plus there’s a great story forming around Actuate’s 3 new business units, an executive team I’ve known and trust (for almost 20 years), and a marketing team with the resources in place to build something special.
What’s on my plate for the first few months? In addition to exploring ways to drive market awareness for the company’s BIRT Analytics offering, I’ll be leading the design of a new company-wide content marketing “hub” concept that leverages my work in this area over the past few years, and will be working my industry contacts for new research partnerships and growth opportunities as well.
Oh, I’ll still be writing, speaking, and spreading the word about the power of small data and personalized analytics. And I also plan to expand the content (more posts!) and contributor roster right here to leverage Small Data Group’s position as the top ranked site for “small data” on Google.
As always I truly appreciate everyone’s feedback, encouragement, and support!
One of the best parts of my research over the past several months as I worked on DCG’s “Bringing the Power of Big Data to the Masses” study (report now available here) has been getting to sample both current viewpoints as well as some of the literature and developments from the past 20+ years on the topic. In fact, big data and today’s analytical tools are very much a product of the “generations” of data and data processing that came before them.
Here’s a cool illustration we created for the report that highlights some of the key developments, deals, and publications related to big data, analytics, and small data since the invention of the web in 1989. We had to leave out a number (many) of potential items, so I’d love to hear what milestones would you have included – enjoy!
With a flurry of new articles on small data in various publications recently – including Wired, Technorati, and AllThingsD – there’s a growing number of voices contributing to the small data movement (a great thing!). But with these new perspectives, I feel like it’s a good time to re-ground the conversation in an actual definition of what we are talking about when we refer to “small data” – or at least what I’m talking about!
So in this post I wanted to share my first cut at a proper definition. Yes, I’ve framed the pillars for small data in many other places, going back to my first guest piece in Forbes on the topic a year ago (!), and more recently I’ve embraced the idea of describing small data as “the last mile of big data.” But these were descriptions or principles vs. definitions for the most part.
So, after spending the last couple month working on Digital Clarity Group‘s new multi-client study on “Bringing the Power of Big Data to the Masses” (sponsored by my friends at Adobe, Actuate, HubSpot, and Visible), seeing how marketers are looking to make analytics more accessible and actionable – and creating some new starter use cases, an interesting thing happened: a definition emerged!
In fact, while our final report is still a week or so away from being available from DCG and our sponsors, I shared the definition with our audience at the Digital Pulse Summit this past week during my panel on small data, and given the response, I wanted to provide it here as well.
A New Definition for Small Data
Small data connects people with timely, meaningful insights (derived from big data and/or “local” sources), organized and packaged – often visually – to be accessible, understandable, and actionable for everyday tasks.
Note, as we describe in our report, this definition applies to the data we have, as well as the end-user apps and analyst workbenches for turning big data set into actionable small data. The key “action” words here are connect, organize, and package, and the “value” (the 4th V of big data) is rooted in making insights available to all (accessible), easy to apply (understandable), and focused on the task at hand (actionable).
In fact, I hope it’s as much a mission statement, as it is a definition. What do you think? Did we nail it?