Friday, August 25, 2017

Dynamics 365, Azure webjobs, Azure Service Bus queues and deadletters

When you need to integrate Dynamics 365 (online) with other applications/services in on premise or in the cloud there are a number of challenges. One of the most commonly used integration patterns is messaging since it allows for loose coupling and it provides a reliable way of integration since both applications don’t necessarily have the same responsiveness or uptime. One of the key extensibility points from an integration perspective that you can leverage is the fact that you can connect Microsoft Dynamics 365 (both on premise and online) with the Azure Service  (See Azure integration with Dynamics 365). In an asynchronous messaging pattern, you will however still need a message processing component.  Since you are already leveraging cloud components, it might make sense to also use Azure for this message processing component.


There a number of Azure components you might take a look at:
  • Azure cloud services: use an Azure worker role for message processing (for some background on this read Should I use cloud services or something else?)
  • Azure app services: use  Azure web jobs for message processing.
  • Azure virtual machine hosting the processing components
  • Azure functions  - similar to Azure web jobs since it was built on the same code base as webjobs and has a similar API.
  • Azure logic apps
If you look at the two first options (worker roles and web jobs), you will notice that they offer similar functionality payload – ability to run code repeatedly. However, there are some differences between the two - primarily in terms of environment customization. Web Jobs are good for running small tasks (not very computation heavy) and useful in scenarios where you do not have the need to depend on host OS features. Worker Roles can be used to run computation heavy workloads and their host OS can be modified a bit (by installing dependency components in OS through startup tasks). Worker roles are costly (similar to running a dedicated VM) while web jobs do not carry any additional cost. In case you need extreme customization on OS environment, then you would simply run a windows service inside custom OS image in Azure VM.  So basically you are looking at a trade off between control (and full flexibility) and agility combined with ease of management.



Now, if you look specifically at Azure webjobs, you will see that they can be triggered by a schedule, messages in Azure storage queue or blobs added to Azure storage and http call by calling the Kudu Webjobs API. But given the fact that Dynamics 365/RM already provides integration  with Azure Service Bus, this is the one that you will typically use – for a good walkthrough take a look at  How to use Azure Service Bus with the WebJobs SDK .

One of the things to keep in mind though, is that when an Azure webjob encounters an error in processing a message, it will retry processing it a specified number of times.  The number of retries is configured through the JobHostConfiguration.Queues.MaxDequeueCount property – see Azure webjobs and JobHostConfiguration for more details. This mechanism has been been built to avoid that a queue-based application gets stuck in a loop receiving and aborting the same message it can not process – this is also referred to as poison message handling.  So if this count is exceeded, the message will be move to the deadletter queue (when it is configured).

But the Azure service bus queues also have a property called MaxDeliveryCount (See QueueDescription class) which is by default set to 10 – so if this setting is lower than the setting in the JobHostConfiguration, your message might be moved to the deadletter queue (DLQ) earlier – see Overview of service bus dead-letter queues for more details.
If you open the deadletter queue with Azure Service Bus Explorer you will notice that these messages have additional properties called DeadLetterReason and DeadLetterDescription – these might be helpful in determining what causes the messages to end up in the DLQ. You can also move a message to the dead letter queue yourself by calling the BrokeredMessage.DeadLetter method.

Lessons learned:  If you are building an integration using Azure Service Bus queue, don't forget to think about how to handle messages which are deadlettered.


Technorati Tags: ,,,,,

Sunday, August 20, 2017

Dynamics 365 monthly reading August 2017


Technical topics (Configuration, customization and extensibility)

Topics for business analysts, power users and end users

Thursday, August 10, 2017

Voice of the Customer for Dynamics 365

In March 2015, Microsoft announced the acquisition of Mojo Surveys – a customer feedback solution which was built solely for Microsoft Dynamics CRM (both online and on-premise).  This product was re-engineered and rebranded to Voice of the Customer (VoC). VoC was initially made available in preview in January 2016 – Preview feature: Voice of the Customer surveys and became generally available in the December 2016 release (VoC solution version 8.3.x and higher). At the moment VoC is only available for Dynamics CRM/365 (online) and apparently there are no plans to make it available for integration with on premise Dynamics CRM/365.

Voice of the Customer adds rule-based, multi-channel feedback surveys integrated with workflows, campaigns and e-mails on top of Dynamics 365. These surveys can be used in a range of scenarios from service feedback to focus groups to sales scenarios. Voice of the Customer allows you design and publish surveys from within Dynamics 365, distribute them to your target audience. Respondents can take part in a survey in different ways, depending on how you sent the invitation out, whether you want anonymous or non-anonymous responses and whether you want to integrate data from CRM into the survey. Responses can be incorporated into a business process by creating response routing rules. Voice of the Customer is available for installation as a preferred solution from the Applications tab in the Dynamics 365 Administration center.


From Voice of the Customer version 9.0 onward, you immediately receive a confirmation screen for accepting the license terms and for activating VoC when configuring it from within the Applications tab.




In previous versions, you first had to trigger the installation and afterwards go to the solution and accept the licensing terms from within the solution screen.



In the previous version, you could also Retrigger response processing to troubleshoot when you did not see responses coming in see Where are my Voice of the Customer responses? for more details. It is unclear where this setting has gone in version 9.x and higher.

Before you install the solution it is important to realize that behind the scenes Voice of the Customer uses a number of different Azure components such as Azure blog storage, Azure web and worker roles, Azure Keyvault, Azure SQL  see Install the Voice of the Customer solution for more details about this. These components are subject to other privacy and compliancy governance policies – for more information see Azure Trust Center

It is quite easy to get started with VoC and the steps are thoroughly outlined in Create a Voice of the Customer survey to get customer feedback . In previous version, I noticed that sometimes the survey definition got corrupted so it is important to quite regularly click on “Preview survey”. If you get an error like “Isv code aborted the operation”, it is best to delete the last question that you added, save the survey, run the preview and re-create the survey question.



Other good walkthroughs you can take a look at are A Step by Step Guide to Create Your First Survey with Dynamics CRM 2016 Voice of the Customer as well as Voice of the Customer for Dynamics CRM 2016 (configuration screens have changed a little bit from 9.x onwards but these articles are still worth taking a look at). One of the things, I also liked about the surveys is the fact that they are also mobile-friendly





Wednesday, August 09, 2017

Managed metadata column values not visible in SharePoint view

I recently encountered a problem with views in a SharePoint site collection – the values of the managed metadata column were not visible anymore for most users in the list views. However  when the user edited the item or viewed the details of an item – they could see the value. Apparently this issue was caused as a side effect of migrating Windows users to claim users. For performance reasons, managed metadata (or taxonomy items) are stored at site collection level in a hidden list called TaxonomyHiddenList which is accessible at the root level of a site collection e.g. http://servername/sites/sitecollection1/lists/taxonomyhiddenlist.  The list by default has read access to all Authenticated users but when performing the claims migration the security was mixed up, by giving all authenticated users read permission again, the issue was resolved.


Thursday, August 03, 2017

Free CRMUG Belux webinar – What’s new in Dynamics 365 Spring 2017 release

By now, you have probably all read about the upcoming Dynamics 365 spring 2017 release. If you want to learn more about it register for the free CRMUG Belux webinar – What’s new in Dynamics 365 Spring 2017 release . This June, Microsoft announced the next Dynamics 365 release, called the “Spring 2017” release wave. During a 3-day (!) long preview session, Microsoft’s product team took a deep dive into the new capabilities that are coming to the platform, such as:
  • An updated user interface, providing a uniform experience for users across web, app and mobile,
  • New features in Dynamics 365 Customer Insights that leverage data across and beyond Dynamics CRM and machine learning capabilities (Also check out Introduction to Dynamics 365 Customer Insights to get started)
  • New social selling capabilities, a direct result of the LinkedIn acquisition
  • New features around scalability and extensibility, -
  • The release of the Business Edition, a sales and marketing app triggered towards small and medium businesses.
Your CRM-UG BeLux team has managed to bundle all these exciting announcements in a 1-hour lunch webinar for you on August 28th. Our presenters, Microsoft’s Philip Verlinden and Dieter De Cock, have even promised there will be time for Q&A.

Introduction to Dynamics 365 for Customer Insights (DCI)

Dynamics 365 for Customer Insights (DCI) was introduced in December 2016 and has recently received some interesting updates in the Dynamics 365 Spring update.  DCI is an Azure-based SAAS service which enables organizations to bring together transactional, web and social customer data to visualize it in a Customer 360°. DCI also allows you apply advanced data transformation algorithms (KPI calculation and machine learning) on this data to generate role-specific insights for different roles within an organization. 



The key differentiators within Dynamics 365 for Customer Insights are:
  • It provides a scalable data infrastructure which can handle millions of interactions, entities and relationships between entities. Building such an infrastructure can be a daunting task and will consume a lot of upfront investment in  IT capacity. By providing a platform where you only pay for the data that you need to handle, you will be able to shift investment to resources and more value added tasks such as integrating the intelligence within your existing business processes
  • Out of the box connector for Dynamics 365 for Customer Engagement (CRM) which allows you to rapidly get going starting from the existing effort that you put into building your CRM platform
  • Ability to match customer identities in different systems using conflation algorithms and predictive matching – this allows to merge/cluster customer identities at scale.
  • Built-in advanced machine learning applications such as segmentation – which identifies cohorts of users with the same characteristics or at the same stage of the customer journey – and built-in predictive scoring. Business users will be able to define the business outcome they want to predict (e.g. customer churn = customer for which status becomes inactive) and DCI will automatically create a machine learning model and validate it (behind the scene it will do feature selection, machine learning selection, training and validation,…). Prediction scores will be automatically generated for new entities or they will be recalculated for entities for which one of the signals/characteristics impacting the predictive score will change. This allows you to get up and running quickly with a machine learning solution without hiring data scientists.
  • Extensible platform – ability to ingest data from multiple data source by building your own connectors and ability to expose DCI entities,metrics and predictions through Power BI, Power Apps or using the REST API
Customer Insights is leveraging the same components which are also used within the Cortana Intelligence Solutions but they are taking it a number of steps further by providing out of the box connectors, configurations screens, ingest and egress paths as well as built-in visualize layer. You however have less control on the inner workings of the Customer Insights solution since this is a SAAS solutions whereas most Cortana Intelligence Solutions combine a SAAS, IAAS and PAAS platform components.



One of the customer cases that you should definitely take a look at is Marston’s and Customer Insights - Marston, is a brewing company operating more than 1700 pubs in the UK and was one of the early customers that started using this application.  In order to create a personalized pub experience for their customers and achieve a more targeted marketing they wanted to bring together customer data from social media such as Facebook, Twitter, and TripAdvisor. They also wanted to look at Wi-Fi registration data, point of sale transactions, table bookings and survey results to understand their customers.  This required a level of data integration and analysis that goes beyond just CRM alone and that’s were Customer Insights comes in. Shown below is an example of dashboard within DCI, which is tailored towards a house manager of a Marston’s inn.



Word of advice: Customer Insights for Dynamics 365 is currently in preview – available functionality is subject to change and both pricing and product naming might change before the final release. From a licensing perspective, we know see two tiers as described on the Dynamics 365 Customer Insights pricing page. Pricing is based on the amount of data which needs to be processed as well as KPIs used, predictive match policies and predictive score models.

References:




Tuesday, August 01, 2017

Introducing Cortana Intelligence Solutions

It has been a couple of months since I last looked at Cortana Intelligence Solutions (CIS) in the Cortana solution gallery so I was pleasantly surprised by a number of new solution templates which have been added.  These solution templates target different problem domains in different industries and are leveraging the full breadth of the Microsoft data platform offering – some of the scenarios covered are:
  • Predictive Maintenance for Aerospace - this Predictive Maintenance solution monitors aircraft and predicts the remaining useful life of aircraft engine components. Uses Azure HDInsight, Azure Machine Learning, Azure Event Hubs, etc … the different components are outlined in a solution diagram which is provided by the solution template as well.
  • Campaign optimization with SQL Server 2016 - demonstrates how to build and deploy a machine learning model with SQL Server 2016 with R Services to recommend actions to maximize the purchase rate of leads targeted by a campaign.
  • Vehicle telemetry analytics - this solution demonstrates how car dealerships, automobile manufacturers and insurance companies can use the capabilities of Cortana Intelligence to gain real-time and predictive insights on vehicle health and driving habits.


Cortana Intelligence Solutions is just one of the pieces in the Cortana Intelligence Suite next to Power BI, Cognitive Services, Microsoft Bot Framework and a whole lot of Azure cloud components such as Azure Machine Learning, Azure HDInsight, Azure SQL Data Warehouse, Azure Data Factory and many more.


Cortana Intelligence Suite provides both a platform and  process guidance (Team Data Science Process) to perform advanced analytics from start to finish. The main goal of Cortana Intelligence Solutions  is to reduce the complexity of deploying advanced data analytics solutions by providing Azure building blocks to operationalize your data science process. By providing these building blocks, Microsoft tries to enable companies to make smarter decisions at a rapid pace without having to worry about the complexity of designing, deploying and operating scalable data architectures.
I truly believe in the potential of data to be able to truly transform a business, a point which has been backed by the Capturing the $1.6 Trillion Data Dividend  white paper which showed that data leaders have a competitive advantages over their peers. These data leaders have a number of characteristics in common in the way that they invest in data:
  • Leverage new data such as linking transactional data with customer behavior, sensor data, social data, mobile data eetc…
  • Expanded use of analytic techniques – including more predictive analytics and big data processing techniques such as Mapreduce
  • Use new metrics – e.g. new ways of looking at operations or measuring performance
  • Include new users – expand the number and type of users who have access to the organization’s data and analytics
The interesting part of the Cortana Intelligence Suite is that it also describes the Team Data Science Process lifecycle which provides a recommended lifecycle which can be used to structure data science projects (see Data science for the rest of us for a good starting point of your new to the field of data science).



A key point that I can’t stress enough is the fact that you need a solution understanding of the business context. Data science projects are typically not IT-driven, business should be in the lead. Data science is a team effort where close cooperation with business is required to understand and identify the business problems. The most complex part of the data science process is formulating the  questions that define the business goals and that data science techniques can target. Most companies agree that managing customer churn is important, but how you define churn is different in each industry and also the impact of churn of specific customers might be different e.g in telco customers are influenced by both friends within the network and friends of friends (See Using social network analysis to predict churn which used data from a Belgian telco provider as well as the research paper Mining telecommunication networks to enhance customer lifetime predictions)

References: