New All-Day Session: Designing Modern Data and Analytics Solutions in Azure

At the fall 2018 PASS Summit in Seattle, I'm excited to be co-presenting a full day pre-conference session with my good friend & fellow BlueGranite colleague Meagan Longoria

Why Do a Pre-Con?

MelissaCoates.JPG

I have a natural inclination to share information that I have learned. Being a hands-on techie is something I absolutely love, but I have a bit of educator in my blood as well. And, continually learning new skills is at the core of what makes me happy. All of which means that I aim to teach others in a way that I would want to learn. 

What Will You Learn? 

This session will very much be about planning the architecture and factors around decision-making, presented in a very practical and realistic way (full abstract can be found here). We will build the components for one reference architecture, using scripts that we will provide you. 

PreconTime.jpg
 

The full abstract can be found on the PASS Summit site. To highlight just a few of the topics that you'll hear about:

  • Going to the cloud - What's easier? What's harder? What trade-offs can you expect to make with respect to cost, control, complexity, performance, and security?
  • Cloud design patterns - In what ways are cloud design patterns different from traditional on-premises solutions? How does that change the typical roles for developers and administrators?
  • Schema-on-read - In what scenarios does schema-on-read work extremely well? In which situations is it not ideal?
  • Patterns-based development - What automation techniques can save you time, improve efficiency, and reduce the chance for error? 
  • Architecture - What does a BI/analytics/DW architecture look like if we value the concept of polyglot persistence vs. architectural simplicity? What kind of differences should we be aware of if we are using a distributed architecture? What are the Azure options for supporting data science and self-service BI?
  • Data storage - When do we want to analyze data in place vs. move it to another data store? What technology options do we have in Azure, and what factors do we want to consider for deciding between data virtualization and data integration? In what cases can you take advantage of a data lake in your architecture? 

Who is the Target Audience?

The ideal audience member has some experience as a data engineer, BI professional, or database developer, and is in the early stages of migrating or building solutions in Azure.

This session is broad because the data platform offerings in Azure are broad with many choices and considerations. Our day job *is* planning and building data solutions in Azure. Meagan and I are very excited to help you get started with building a solid data architecture in Azure.

More details and to register: Designing Modern Data and Analytics Solutions in Azure

Third Annual SQL Saturday BI Edition in Charlotte

The third annual SQL Saturday, BI Edition, is back in Charlotte! It's coming up next weekend, on Saturday, October 4th.  

SQL Saturday is a free* training event for Database and Business Intelligence professionals. It's a conference condensed into a single day. The event includes training sessions led by local, regional, and national speakers, and lots of opportunities for networking and meeting up with peers. There's also exhibitor tables with useful products, information, and job opportunities.

The schedule has 40 technical sessions, at varying levels, on:

  • Data Analytics
  • Cloud
  • Big Data
  • Information Management
  • Development
  • Information Delivery
  • Environment Architecture
  • System Administration
  • Professional Development

More information: http://www.sqlsaturday.com/330/eventhome.aspx 

Registration: http://www.sqlsaturday.com/330/register.aspx

Schedule: http://www.sqlsaturday.com/330/schedule.aspx 

I hope to you see you there at SQL Saturday next weekend - be sure to say hi!

*If you would like to join us for lunch, we ask that you pay $10 to help us cover the cost of feeding a few hundred people. However, if you'd rather bring your own, that's ok too - just go through the registration process like normal. When it asks you to pay for lunch at PayPal, go ahead and close that window.

Live Blog - Keynote 2 - 2014 PASS BA Conference with David McCandless

The day 2 keynote at the PASS Business Analytics Conference is with David McCandless.  I'm thrilled to be part of the live-blogging group, so I'll do my best to communicate what's happening for those who couldn't join us in San Jose.

About David McCandless.  David (Bio | Blog) is an information designer, author, and data journalist from London.  He writes for The Guardian sometimes, which has a great Data Store of journalism articles + accompanying data to download.

Live blog of the keynote

Denise McInerney

8:06 - Welcome for David McCandless.

David McCandless

8:08 - Hello!  Has a passion for collecting and organization of data.  Gathering the data and visualizing it in graphical images anyone can understand.  Strange and interesting and imaginative things can happen.

8:10 - Billions of numbers of dollars. These numbers are routinely circulated but are too big to get our heads around. Shows the Billion Dollar-o-Gram. American people give over $3 Billion a year to charities. It's the connections between the data that make it interesting.

8:11 - How to bring this data down to where we can relate to it?  Converted some of the same numbers to a metric we can all relate to.  How much taxpayers pay into each area per day. Goal is so we can more readily relate to it.

8:12 - Playing an animation called Debtris.

8:14 - Love to play with data, have fun with it, see where it leads.  Tries to use it for storytelling & finding underlying patterns.  Showing a timeline of the world's biggest fears over the years. Then shows a pattern for violent video games - there's a regularity to it with twin peaks every year in November & April.  The Columbine shooting has been linked to violent video games. These kinds of patterns won't emerge in a spreadsheet; you have to visualize it to see it.

8:17 - There are stories and patterns lurking in vast data sets to be used and examined.  Showing a new visualization that shows two big peaks per year & mini-peaks each Monday. Asking audience who has seen this visualization before. Turns out it's most common break-up times. Includes spring break, april fools day, summer, and right before holidays. This is according to Facebook status updates.

8:20 - Phrase that "data is the new oil." It's a resource.  David would adapt it to be "Data is the new soil" in that it's a material to get your hands dirty with & nurture & bloom.

photo 3.JPG

8:21 - The term Big Data is confusing because it's both a noun and a verb.  Big Data is many processes - gathering, handling, structuring, examining, discovery, and delivery. Many companies circulate in the top regions.

8:22 - Recently David was investigating if horoscopes all say the same thing?  Did a word frequency analysis. Used Yahoo's horoscope database. If you retrieve from Yahoo's database too frequency they'll ban your IP address for a week so they had to find the right frequency to extract the data.

8:23 - Ended up boiling the horoscope down to several general statements about people being happy.

8:24 - You can learn a lot of the skills you need just by playing around with the data.

8:25 - Recently looked into who has the biggest military budget?  It's the US by far.  Ours is > $700 billion. But...is it true that America has the world's biggest budget?  To be fair we need to ask, who spends as a proportion of their GDP?  Then you get a very different picture where USA drops down to 6th.  The context is what provides meaning.

8:27 - Who has the biggest army?  China with > 2.1 million.  But if we normalize the figure and compare to the population, then China falls to 124th and N. Korea comes to the top. US is 45th.  Need context and meaning to get a clearer, truer picture.

8:28 - Hans Rosling has a phrase: "Let the dataset change your mindset."  Use it to view the world more accurately.

8:30 - Looking into the # of communicating civilizations in the galaxy.  Result is 46. There are 6.9 trillion galaxies in the universe.

8:32 - What is dataviz good for?  Reframing, finding patterns, analysis, normalizing, contextualizing, and pimping.  

8:33 - Looking at his visualized CV to show his work experience visually.

8:34 - David has never been to design school, never trained. Just likes to pick things up & learn by playing with it. Felt like he had an innate sense & become design-literate.

8:35 - Every day we are looking at the Internet. This is training us to look at data in design terms. Now if we visit a shoddy website, how little we trust it? 

8:36 - We can process visualization many times faster than text. In data visualization, it's more memorable & impactful.

8:37 - Showing a visualization of who is suing who in telecoms.  If you remove labels, your eye can focus on answering questions with size & color. Then layer on the story.

8:38 - Metaaphorically, using dataviz is like a new kind of camera.  

8:39 - Looking at a visualization of drug use by country. Good laugh about the Aussies.

8:40 - Showing most popular search terms in a 'painted' visualization. More interesting to look at the information map.

8:41 - Showing a venn diagram to look at what pigs, birds, and people have in common?  Flu.  He calls this the "Infl-Venn-Za."

8:42 - Looking at a dataviz - if Twitter community were 100 people. Then turns to a dataviz about TV watching.

8:45 - Showing at a dataviz of the most commonly used PIN numbers. Reveals habits people have.

8:46 - Showing graphic of causes of death which was done for an art gallery. 

8:47 - Goal is to 'see the invisible' in the data.

8:48 - Looking at a dataviz of supplements. Relationship between efficacy and popularity? This one graphic took two months to analyze and create. All this data is in Google Docs and open.

8:49 - You can condense a huge amount of data into a small space. Then enjoy it as if it's a painting.

8:50 - Looking at biggest data breaches.

8:51 - The active exploration & joy of interactivity.  Interactive apps for storyfinding is the new frontier. It reveals the invisible.

8:52 - Qualitative can be visualized too. Showing an infographic of left & right government to compare & contrast the worlds. The same word, like equality, can have very different meanings.

8:56 - David puts his failed graphics on the web for people to learn from.  An overly complex graphic just moves the problem. The graphic needs to unveiled. 

8:57 - He tends to create circular diagrams, but their usability. It's beautiful, but what is going is on?  

8:58 - An infographic that fills every possible space can be too overwhelming.

8:59 - A "charticle" is one that is part chart/part article. What can be removed? What works?

9:00 - What does work? Information and Design. A good visualization combines many of these things.

9:01 - Allowing yourself to play with data, get your hands dirty, helps you see what comes out. In that spirit, David wants to play a little game:  the dataviz quiz.  He has removed the labels & is asking the audience to say what the data is representing. Some good laughs!

9:09 - Q&A with the audience.  All of his datasets are available on Google Docs. He and his team spend 80% of their time on the data, and only 20% on the visualization.

 

 

Live Blog - Keynote 1 - 2014 PASS BA Conference with Kamal Hathi and Amir Netz

The day 1 keynote at the PASS Business Analytics Conference is with Kamal Hathi and Amir Netz.  I'm thrilled to be part of the live-blogging group, so I'll do my best to communicate what's happening for those who couldn't join us in San Jose.

About Kamal Hathi.  Kamal (Twitter | LinkedIn) is responsible for managing the overall strategy, design, development and delivery of SQL Server business intelligence technologies. He's been with Microsoft since 1996, first as as Program Manager for MSN, then a Product Unit Manager for SSAS and SSIS, and most recently as General Manager for SQL Server BI technologies. One thing Kamal believe strongly in is data stewardship. To hear more about this and other thoughts on where things are headed in the BI space, check out Mark Vaillancourt's interview with Kamal at last year's PASS Summit: http://markvsql.com/2013/11/pass-summit-interview-with-kamal-hathi/.

About Amir Netz.  Amir (Twitter | LinkedIn) is a Microsoft Technical Fellow and chief achitect for Microsoft's BI technologies. Amir came to Microsoft in 1997 from Panorama Software, first as a Partner Architect, then Distinguished Engineer, prior to Technical Fellow. One thing Amir advocates for is democratization of BI, by empowering users through self-service. A Technical Fellow is Microsoft's highest technical leadership rank, afforded to only about 20 people.  

Will we hear some exciting announcements from Kamal and Amir? I hope so! Either way, their keynote should be informative and entertaining.

Live blog of the keynote

Tom LaRock (PASS President)

Source:  PASS BAC Flickr acct at    https://www.flickr.com/photos/passevents/sets/72157644424706732

Source:  PASS BAC Flickr acct at  https://www.flickr.com/photos/passevents/sets/72157644424706732

8:03 - PASS President Tom LaRock takes the stage.  Gets a round of applause when he asks, "We get paid to work with data every day. Who is excited about that?"

8:05 - Who are we? Variety of titles. In this room we have over 300 industry-leading companies. Since 1999 PASS has supported data professionals. PASS runs through the involvement of hundreds of passionate volunteers - they need your help.   PASS now has over 100,000 members.

8:06 - Think about what you are passionate about with data & figure out a way to get involved with PASS. Lots of opportunities, whatever you are interested in, there's a way for you to be involved.

8:07 - Next 2 days there are over 70+ sessions + Community Zone. Running group tomorrow too, sponsored by SQL Sentry. Morning breakfast sessions. Unconference tonight. A lot is going on! Reminder that all sessions are being recorded & available on USB.

8:08 - Thank you to partners. Give sponsors a hug!

8:09 - Next conference April 20-22, 2015 in Santa Clara, California.

John Whittaker (Dell)

Source:  PASS BAC Flickr acct at  https://www.flickr.com/photos/passevents/sets/72157644424706732

Source:  PASS BAC Flickr acct at  https://www.flickr.com/photos/passevents/sets/72157644424706732

8:10 - Welcome to John Whittaker, Sr. Director Information Management, from Dell Software. Speaking about Big Data, Predictive, & The Middle Market.  A key component Dell builds solutions for is the middle market. Is Big Data real & happening in the middle market?  How should the middle market approach Big Data?  Not just for large enterprises. 96% of organizations surveyed are in-flight with an existing project or plan to start one in 2014. 

8:13 - #1 reason for success of a Big Data project:  Strong cooperation or collaboration between business and IT. Need LOB there to guide the outcome. Rogue outcomes around Big Data can happen when data governance isn't present.

8:14 - Overwhelmingly there were two primary responses for doing Big Data:  (1) real-time processing of data & analytics, and (2) predictive analytics.  

8:15 - Biggest challenges for Big Data projects:  (1) Complexity of new data types & structures, and (2) Sheer volume of data slows processing. Followed by (3) Budget limitations.  But really...it's all about data complexity.

8:17 - Other Key Takeaways:  (1) Skills gaps (skills & tools).  Tooling is unsophisticated. (2) Decisions should be split evenly between IT and LoB business teams.  (3) Is it making a difference?  50% of orgs with a big data initiative iflight are satisfied with quality & speed of their decision making.  

8:20 - Dell has over 30 years of information management software experience. Dell's goal is to offer end-end solutions for data management. 

Kamal Hathi & Amir Netz (Microsoft)

8:23 - Amir & Kamal introduce themselves. Been an amazing year since they last spoke to this audience. A year of innovation:  Not just lots of new technologies but also a new CEO. Never had a CEO before announcing a new version of SQL Server.

8:24 - Kamal is jazzed because Satya Nadella has made Indian accents cool again.  Lots of laughter.

8:25 - 2 million+ downloads of Power Pivot since 2010. 100,000+ downloads of Power Query since Feb. 2014. 55,000 Power Map preview downloads. 100 million+ HDInsight compute hours. 12,500+ Power BI tenants activated.

8:26 - Kamal is going to use Power BI to look at Power BI.  He's using Q&A to ask how many Power BI tenants. Explains what a tenant...Microsoft is 1 tenant with 100k users. Company has one tenant, many users per tenant.  Asking Power BI tenants by country. Seeing map of global users - biggest in US. It's in Bermuda to Iceland and everywhere. 

8:28 - Kamal looking how Power BI tenants are growing.  Line zooming up over last 2 months. Spike when Q&A went public in September. Another spike in February when the GA announcement went out. Nice growth happening so far.  

8:29 - Number of questions answered by Q&A last month:  1,091,547 - that's over a million!  They look at this so they can understand the features and who is using them and how they can be improved.  

8:30 - Looking at telemetry - the most used feature of Q&A is choosing the "select suggestion" drop-down. They know to continue investing in this.

8:31 - Share URL by E-mail is new & is just taking off. Looking at other Data Discovery features, like changing fields displayed or filters to see the visualization change.  Usage for some of these types of features have doubled in the last month because Microsoft has made these features more prominent and discoverable - because it's there by default more people are using it.

8:32 - Amir says looking at the stats is interesting, but wants to look at it in action instead... They asked for people to show how Power BI is used. Announcing the winner of the Power BI demo contest:  Michael Carper.  Brings Michael onto the stage.  Amir is animatedly recapping what Michael's demo is about - can you affect the results of an NBA game by tweeting?  If you tweet in the 3rd quarter you have the best chance of affecting the game!  More tweets, more wins!

8:35 - Amir is presenting (another!) award to Michael for his Power BI Demo Contest entry.

8:36 - Amir explains how a cloud service like Power BI is a game-changer for Microsoft. They learn from the data & can improve the service every single months.  Kamal showing usage by browser usage. HTML5 running across various browsers.  Kamal is running a Chromebook & explaining how the web-based display works the same across devices.

8:38 - Amir says they hear the feedback: it should always work on every platform.  "BI on any device." 

8:38 - Announcement from Amir:  iOS app for Power BI will be available in the App Store by this summer.  The native app experience will be the best experience.

8:39 - Microsoft knows we have many in-premises investments...First step towards this is the scheduled data refresh in Power BI.  Kamal bringing up a Power BI site to see the thumbnails. These thumbnails are different....they are actually Reporting Services and the data was on-premises.

8:41 - Announcement from Amir:  SSRS will be a native component of Power BI.  All the functionality of SSRS will be available in Power BI by this summer.  

8:42 - SSRS connectivity will be real-time connectivity to the on-prem data source. No refresh required.

8:43 - Kamal explaining they want to promise companies will be able to choose cloud or on-premises, whichever works for you.

8:44 - Amir reflecting years ago that BI was just about the "guy in IT" or "the consultant" and we trusted the professional will do a great job & deliver value to the end users. Next was the age of self-service BI - the analyst or power user should be able to able to create or consume. Still will be some who create and some who consume.  Now we are entering another new age:  the age of Data Culture.  You don't have to be an analyst - anyone who has a question or is curious should be able to answer their own question. Goal is to make the tools so simple that people can do this.

8:46 - Amir loves music and movies-he's done both. This time going to do a demo on Hawaii, which he loves.  Hawaii regularly asks people via surveys how they spend their time & how they spend their money. Amir realized Hawaii was collecting all these forms so that he could have a great Power BI demo.  =)

8:47 - Amir is using Q&A to look at average spending per day.  Is about $182 per person. Then a breakdown by island. Lana'i is quite a bit higher than the rest - just over $300, followed by O'ahu at $200.  

8:49 - Amir looking at # of visits plus where people are coming from. The big spenders seem to be from Japan. Now looking at visits by date and seasonality. Canadians don't seem to spend much & come in the winter.

8:51 - Amir has observed lots of public displays of affection from the Japanese. So he, of course, checked the data!  Turns out lots of Japanese come to Hawaii for their honeymoon. He says Japanese honeymooners are higher than any other segment of people. Has a new investment suggestion for us here.  

8:53 - Emphasizing that using natural language and ease-of-use experience to every area of BI, even areas that are boring and dry.  New button at the top of the Power BI site page to create a new KPI.  Brings up a KPI Editor - this KPI editor uses Q&A to get a visualization and choose "add selected tiles."  Just built a dashboard page all using natural language.  These are displaying across the top of the Power BI site page.

8:55 - Now Amir wants to look at how predictive analytics can be so easy that every user can do it?  

8:56 - Announcement from Amir:  Forecasting in Power View.  Time-series analysis.  [Big round of applause.]  Data scientists & user-interface designers have been working on this for a year. Goal is for someone without that background to be able to do predictive analytics.

8:58 - Forecasting is available today.  Available on every line chart.  Drag the line forward and the future points will appear. There's also a pane on the right to make some modifications to it.

8:59 - Looking at a one-time event in the line chart that is skewing the future line chart results. Demonstrating changing this point that's an outlier so that it will then adjust the future results.

9:00 - How to know confidence a year ago?  Can check validity of results by looking at if it were done, say a year ago, what would it have said?  

9:01 - Forecasting needs a time dimension, a measure, and a line chart for it to work.

9:02 - Pulling a volunteer from the audience, Julie, to drive the demo without knowing ahead of time what she will be doing. 

9:03 - Showing a standard Power View report on the large perceptive pixel device. What about users who have questions that haven't been answered by the report that is published?

9:04 - Announcement from Amir:  Data Exploration Mode is going to be available to every user. This is a new button at the top right.  Get a field list now. Full power of working with the report to change it on the fly within Excel Online.  

9:05 - Announcement from Amir:  Tree Map visualization.  Shrieks from Jen Underwood when Data Exploration Mode just returned a Tree Map.

9:07 - Demonstrating the ability in Data Exploration Mode to drag data points from one chart to another to affect the visualization being displayed. Not just cross-filtering anymore.

9:08 - Demonstrating creating a new chart by dragging a data point to the canvas. Selection of chart type appears on the side of the new data visualization to select chart type.

9:10 - Dragging one column name from the field list over to the canvas - # of clicks required is reduced.  Demonstrating dragging the "Unknown" data point into a garbage can, to improve the bar chart display.

9:12 - Announcement from Amir:  New "Bump" gesture. Drag two data visualizations into each other, to bump them, and they'll combine into one chart.  Then do a "Shake" (wiggle the chart) to separate the visualization back into separate charts.

9:15 - Wrap up. The Journey to Data Culture Begins Today.