Last week I gave a talk at the excellent SPARK Boston event along with some other marketing heros of mine including CC Chapman and Kyle Lacy. Of course the topic was Small Data, with a brief history of the topic and a drill down into my thoughts about the movement providing key inputs into a new design philosophy. I also teased the new mobile app my colleagues have been building with the smart folks from Raiz Labs – which provides a great use case for small data in action.
You can check out my entire presentation and a few highlights of our new app in my SlideShare below. Enjoy!
Just as embedded processors (and cheap memory) sparked a new generation of smart systems and consumer devices like the smartphone, embedded BI and analytics – enabled by advances in big and small data processing, open source projects like BIRT and Hadoop, and rich APIs – has the potential to change the face of many categories of applications (and devices).
Think hyper personalized and portable customer experiences, or smarter trading grids that anticipate disruptions or automatically seek out the best deal. Or new views into markets or business operations that reveal previously unseen relationships or potential innovations.
Meanwhile we are all looking to get closer to our customers, by gaining a true 360 degree view of what they want and how they are interacting with us and each other. This is where some of the new Analytics-as-a-Service (“AaaS”) offerings like Watson Analytics and the recently launched OpenText Big Data Analytics in the Cloud fits in. Combining advanced and predictive analytics, delivered as an easy to use managed cloud offering, AaaS aims to bring the power of big data to everyday business users, creating one view of their customer base, with a super-fast dedicated analytics data base and pre-built algorithms for handling the most common marketing and operational analyses.
The future of analytics is clearly about these types of tools that serve the growing population of “citizen data scientists.” It’s also about delivering insights from new data sources (think IoT) to users on their device of choice like smartphones, tablets or even smart watches. And building on a foundation of good information design (as detailed by Edward Tufte), blending the right unstructured and structured content, and applying the latest digital engagement models and approaches (like those from my friend and former colleague Esteban Kolsky).
Discover more in Vegas at Enterprise World
All of these scenarios will be front and center in the Analytics track at the upcoming Enterprise World 2015 event in Las Vegas in early November. I’ll be there with a cast of 1000’s delivering a couple keynote addresses as well as parts of several breakouts and demos looking at the role of big and small data, analytics and digital asset management, and helping to premiere our latest IoT demo (hint: it includes a network of Raspberry Pi-based sensors, an MQTT broker, and visuals powered by the OpenText iHub).
Learn more about the event and our sessions by clicking here.
If the desire to make analytics more accessible and actionable for everyday tasks helped kickstart the small data movement, the growing focus on the best ways to access/manage/visualize IoT devices and data offers to shift the small data discussion into hyperdrive. In fact, over the past couple months we’ve seen some great posts and discussions related to why small data is essential to harnessing the power of IoT. Here’s a review of 5 of them, including my latest article on the potential of new types of embedded intelligence. Enjoy!
1. Forbes.com – Forget Big Data – Small Data Is Driving The Internet Of Things – in this compelling, concise piece by Mike Kavis, he talks about the value of a hybrid small (and big) data approach when dealing with sensor data. His perspective:
“Small data knows what a tracked object is doing. If you want to understand why the object is doing that, then big data is what you seek.”
2. Economic Times (India) – EMC bets on ‘small data’ for future growth – interesting interview with EMC exec Guy Churchward, who argues that billions of interconnected devices like driverless cars will create “small data sprawl” – requiring new tools and approaches unavailable today.
3. Enterprise Apps Today – How IoT Will Change Big Data Analytics – in this roundup article there’s a number of scenarios, including a pretty cool section with an AT&T executive who points out the importance of information management:
“…determining what type of data is important, what should be transmitted immediately, what should be stored and for how long, and what information should be discarded. Otherwise, you could end up with an almost infinite pile of data to analyze, when only a relatively small portion is of real importance.”
4. Logistics Viewpoints – The Benefits of “Small Data” in Logistics – it’s always good to sample specialty industry perspectives, and this one features some good insights from ARC Advisory expert Ralph Rio, who talks about a “small data within Big Data” approach.
5. CTOVision.com – Beyond the IoT Buzz Is A New Horizon of Embedded Intelligence, Information Flows and Seriously Smart Apps – my latest op-ed for CTOvision, where I look at requirements for becoming data driven and some of our learnings from working with device data, including the role of new wearable computers as sources AND destinations for small data:
“…consumer wearables and other mobile devices have a dual role in this scenario as they not only create data, but they also consume (and display) it.”
What other articles or perspectives did I miss? I’d love to hear your ideas!
And if you are in London, Munich, Paris, New York or Toronto over the next few weeks and want to see our latest IoT demos in action, my team and I will be presenting at the OpenText Innovation Tour 2015 events – if you are interested in attending, check out the schedule here.
Back in the summer I explored the concept of “wise” devices being proposed by Fitbit designer Gadi Amit and introduced the idea that small data will be the OS for these mobile and wearable devices.
Since then, my teams at Actuate have been exploring these ideas and accelerating our work around applying information design best practices for a new generation of rich mobile apps and embedded analytics. The goal: expand the boundaries and our understanding of what it means to assemble and display intelligence in context. We’ve also been teaming with our engineering group to look at new ways to demonstrate the rich APIs provided by the BIRT iHub to better access real-time device data, visualize it, and embed these packaged insights on “non-traditional” devices like smartwatches, tablets or even large-format displays.
Some initial results – including a very cool IoT-telematics demo that leverages the BIRT technology stack and open data standards to show how to connect your car to your smartwatch – were first premiered by Actuate’s Kris Clark at EclipseCon Europe. And more recently this demo was updated and featured at our ongoing Data Driven Summit events, along with the IoS tablet app shown above which demonstrates rich, fluid visualizations in a mobile BI scenario.
These concepts and Actuate’s overall strategy for the Embedded Analytics market were also part of a media and analyst roundtable during our London event. And judging by the press coverage (see here, here and here), “Wearable BI” is a pretty compelling idea!
Observations from the Field
The great thing about getting into the field and sharing perspectives with practitioners and project owners around the globe (I’ve been in 6 cities on 3 continents over the past 3 weeks, speaking to and sharing with hundreds of attendees as part of our Summit program) is that we have fresh insight into how organizations are looking at visualizing their data, and where they are focusing in 2015 and beyond. As expected, mobile devices and data are a big driver, but so is the idea of operationalizing transactional big data by embedding analytics in more places, and making these insights more consumable by more users by raising the bar in terms of information design.
In fact my colleague Mark Gamble’s session on Visualization and UX Best Practices – featuring the best tips from Edward Tufte and Stephen Few and our own team – has been consistently one of the most popular and highest rated sessions during our recent tour.
Other notes, takeaways, and resources:
- The Small data philosophy is even more relevant today than when I started writing about it 2 years ago in Forbes. More analysts and companies are focusing on the “last mile of Big Data” as the place where value is created, especially when tackling everyday marketing challenges and designing data-driven apps for mobile and IoT. Since my last post some good articles to check out on the topic include these new pieces in Fortune and Forbes , my latest op-ed in CMSWire, and this cool infographic from Constant Contact.
- For data-driven mobile apps, a native/hybrid approach works best. Sure, native is great for games, but if you are creating real business apps, there are benefits of having some native code local to the device, and some non-platform-specific components resident on a back-end server to provide maximum flexibility and performance (and re-use!) for serving up rich visualizations that scale up to millions of users, and scale down to match individual preferences and limitations of small displays. For more detail, I’ve posted the slides from our Mobile and IoT session at Data Driven Summit on my Slideshare.
- Many organizations are looking to bring the power of big data to the masses. In fact this is arguably one of the top priorities for 2015. This means taking a fresh look at ways to turn data into information, and information into embedded intelligence, one of the themes Actuate’s CEO touched on at our Summits. It also means envisioning new use cases and new UXs for mobile (like our smartwatch app) – something BI guru Howard Dresner has proposed, and serving up alerts and contextual visuals that are foundational to the next generation of wearable BI and intelligent apps.
So, what’s on your list of project for 2015? I’d love to connect and hear about your plans.
Also, if you are in NYC on December 3 and want to see our IoT demo live and hear from customers, BIRT experts, and industry gurus about the latest thinking around data driven apps, there are still a limited number of seats for Data Driven Summit – New York. You can get more info and sign up (it’s free) here.
At the beginning of the year it looked like 2014 was shaping up to be a year when Small Data moved out of the shadows and into the mainstream (or maybe even THE Year of Small Data as I predicted in this op-ed for ZDNet). Judging by the volume of articles, social chatter, and casual conversation I’ve had at numerous industry events, Small Data has arrived. But it still means different things to different people. And that’s cool, since there’s a lot we’ve learned over the past couple years about why it’s important – not only as an alternative to Big Data – but also as a design philosophy and movement that shifts the conversation from processing power…to people!
So if you’re new to the topic, or just curious how Small Data has grown up, I’ve collected my “Top 10” articles for sampling the early concepts as well as the latest thinking. There are many more pieces I’ve left off, but this might serve as a core reading list. And yes, I’ve included just 2 of my own articles, but if you like them you can find many more on this blog or on the Actuate buzz page.
The Early Days – Foundations
Forget Big Data, Small Data is the Real Revolution – Rufus Pollock’s (@rufuspollock) original post on the power of Small Data, collaboration and “decentralized data wrangling” from April 2013. Consistently one of the top 5 most cited pieces on the topic, it’s a great companion to my Forbes piece below.
These Smart, Social Apps Bring Big Data Down to Size – my piece with Mark Fidelman (@markfidelman) from October 2012 that introduced the pillars of small data design: make it simple, be smart, think mobile (be responsive) and predicted a new wave of user-centric business intelligence.
What happens when each patient becomes their own “universe” of unique medical data? – Prof. Deborah Estrin (of @cornell_tech) talk at TEDMed 2013 has key ideas around managing our own personal Small Data. Especially relevant to not only privacy discussions but also those pondering the next wave of wearables and fitness tracking devices.
Data-driven Marketing – 3 Perspectives
Small Data Can Help Businesses Be More Human – great post on the need for “human scale” in marketing by Brand Networks CEO Jamie Tedford (@jamietedford).
What the “Small Data” Revolution Means for Marketers – it’s all about delivering more targeted, more personalized and all-around more intelligent marketing campaigns says CommandIQ (@CommandIQ) CEO Noah Jessop. Right on!
What’s The Big Deal About ‘Small Data’? – recent roundup of perspectives on the marketing front (yup, including my own) by my friends at CMO.com.
Is Small the New Big? – Mainstream Thinking and New Voices
Focus on data value, not its ‘bigness’ – one of my recent op-eds on data value, decentralization and the new CRM. Good overview piece if you want to get a high level perspective and stir up thinking within your own organization.
In Praise of ‘Small Data’: How Targeted Analytics — Not ‘Big Data’ — Are Transforming Education Today – new post by Brian Kibby (@BrianKibby), President of McGraw-Hill’s Higher Education Group.
Your Sushi May Be Getting Smarter – new piece in The Atlantic by Tech Editor Adrienne LaFrance (@AdrienneLaF) on “smart objects,” food safety and the power of Small Data (and IoT).
How To Create Incredible Customer Service Through The ‘Small Data’ Advantage – new piece by customer service speaker and author Micah Solomon (@micahsolomon). While a lot of my focus to-date has been on marketing use cases, I think there are some great ones in customer service/loyalty as well as this post suggests.
What articles did I miss? What would you add if I made it a “top 20” list instead?
In a thought provoking interview with CNET published this past week, Fitbit designer Gadi Amit explores the use of wearables in everyday applications – and introduces the notion of “wise” devices that provide just the right information, when and where we need it.
Beyond the fact that Amit’s firm is designing wearables for unique markets like babies (well I guess really for their parents) and pets (!), what struck me in this piece was Amit’s perspective – certainly shaped by his role as president and lead designer at design firm NewDealDesign – on the state of wearables, their future, and our relationship with them. Specifically, in response to a question about how wearables will be integrated into our daily lives, he states:
The interesting thing is when I say that, people immediately jump to the conclusion that we will be cyborgs. My goal with designing this is that we won’t be cyborgs. We actually will become more human and more free from the technology. What we have now in the design business is two camps: there is the camp that wants to create a lot of data and wants to analyse a lot of data; and there is the other camp which I belong to that tries to create devices that are not smart, they are actually wise. They are more than smart, they are wise enough to understand you, to filter and allow you to go on with your life with all their data processing in the background giving you hints of what is essential when it is essential.
Having data processing in the background and focusing on what information is essential is of course very much in line with the small data “aesthetic” we’ve been promoting here and in a number of venues over the past 2 years, so it’s cool to hear validation from another corner. As a former AI/machine learning guy, I also like the idea of “wise” devices that understand context and personal preferences, and can make a case that small data will in fact be the new “OS” for these devices (more in a future post).
But even more so, if we think of the cyborg comment as a challenge to all of us, I think we need to consider the element of “humanness” as we create new apps and digital experiences. And perhaps provide better opportunities and incentives to untether/unplug (partially?) from our digital devices, even as consumers clamor for faster, more personal, more portable, and ultimately more satisfying data-enriched experiences.
Designing Data-driven Apps
Speaking of the new data consumer, I’ve been spending more time with developers and those thinking about the future of customer facing apps, and recently created a talk on design principles that builds on some of the work you’ve read about on this very blog. As always I believe that data-driven design is an art and a science, so it’s been fun to brush up on the science/tech part for sure.
Of course our first job is still to think about the end-consumer, and how we can inform, connect, and motivate them to get involved or take action. As an aside, if you’ve paid attention to how I’ve presented this last point, I’ve always used Nike Fuelband as my example, so with news that Nike is getting out of the fitness hardware business (good analysis in this Gigaom piece), it’s been interesting to see Fitbit and even Samsung step up their efforts ahead of the likely fall iWatch debut.
On the business side, beyond understanding the value of data along the customer journey and focusing on “last mile” functionality, having a scalable foundation that can potentially support millions of users and large data sets from many sources (before it is transformed into useful small data) is essential as we look to bring powerful, yet human-scale, smart (wise) apps to the masses. So is a community to drive innovation – like the 3.5 million BIRT developers, or 600K+ Drupal users and coders.
Many of these ideas (and some examples) were covered in the talk I did with SD Times recently. There’s a link to the replay and a summary by my colleague Fred Sandsmark on the Actuate blog – which you can read here.
I also presented a longer version focused on bringing the power of advanced analytics to “everyday tasks” at the CAMP IT big data event this past week, (a well-produced event by the way) and plan to post those slides to my slideshare shortly.
Finally, I will be moderating a very cool expert panel on “building the next big app” at a special event Actuate is hosting in San Jose on the evening of July 10. Scheduled to join me on stage will be Eclipse Foundation Executive Director Mike Milinkovich, plus industry watcher and enterprise apps futurist Esteban Kolsky, along with 1-2 other special guests. We’ll explore how consumer experiences will (and are) be shaped by new devices and data, open source driven innovation, and next-generation design tools and practices.
Be sure to let me know if you’ll be in the area and want to join us, since I have a limited number of VIP passes to share.
Big Data is everywhere. Yet many companies lack a clear vision for rolling out big data in practical, measured steps. At the same time, with social networking, BYOD, and expectations from interactions with brands like Amazon, Apple, and Nike, most employees have seen the potential, yet many aren’t equipped to harness this power in their workplace. The small data movement — which I’ve been writing about for the past 6 months (and thinking about for many years) — aims to address these challenges and re-envision the “last mile” of big data via consumer-style, more responsive, more social apps that truly turn insight into action.
Not surprisingly, this idea is getting a LOT of attention. In fact, over the past 6 months there’s been as much published about small data as there was in the previous 3 years. Clearly small data has gone big time as a theme, concept, and set of tools.
So why is small data taking off?
- First, while understandably there’s a lot of excitement about big data, there’s a growing reality that doing it at scale and waiting for all the trickle down benefits can take a lot of time. Especially if you’re not in the C-suite.
- Second, (and here’s where I jump on my soapbox), the last mile of big data is really where the value is created, opinions are formed, insights are shared, and action are made. By non data scientists. Everyday.
- Third, there’s some great consumer examples of small data hitting the stage, most notably the much hyped but potentially game-changing Google Glass, which brings together small data and visualization with wearable computing, all in a semi-stylish package.
So who is fueling the small data movement? I wrote about 10 vendors who get the value of enabling simple, smart, responsive, socially aware tools and solutions in my last post. And a couple weeks back I wrote about 5 additional vendors and a new study I’m doing with Digital Clarity Group in a post on the DCG blog, which you can read here.
As small data takes off, I will continue to provide updates and thoughts both here and on the DCG blog. As always I’d love your thoughts and suggestions in the comments below. And if you’d like to arrange a briefing with me and my colleagues at DCG just email me at allen AT digitalclaritygroup DOT com.
Thanks for stopping by!