A common discussion in any project is what integrations are needed? This is a big question and often a point of much debate. I hope that this article helps demystify integration and make you consider your approach. The biggest question being to integrate or not to integrate?
Why do we need to integrate?
The primary question that we need to ask when discussing integrations is why do we need to integrate? By asking this crucial question will help determine the integration pattern that should be implemented. Depending on the reason why to integrate there may be a need to store all the data that has been integrated within Dynamics 365 or the Common Data Service, there are however other instances that will not require the data to exist outwith the source system as it only needs to be surfaced for information purposes. To decide if data should live in Dynamics 365 or the Common Data Service, ask yourself the following questions:
- Do I need to trigger any logic from the data?
- Do I need to be able to trigger action from the data automatically?
- Do I want to create dashboards with the data?
- Do I need to use the data in more than one entity?
If you answer yes to any of the above questions, you likely need an integration that creates the data in the target system. If however, you need to give the users visibility of the data without any data integration, you can use a pattern that surfaces the data.
Other important considerations are what is the impact of each integration pattern. The most significant factors are around data volume and storage from a technical standpoint and compliance and regulation from a business standpoint.
Several typical design impacts should be understood.
Integration Patterns
Integrations are broken down into three key areas.
User Interface Integration
In a user interface integration, two applications are integrated so that a user can carry out an operation that involves two different applications – without having to take into account that he or she is running two applications.
Seamless user interface integration
Seamless user interface integration can be used to present the functionality as totally integrated with Dynamics 365. The result is two different applications seeming to be one total system, even when the work processes go back and forth across the systems. One example of seamless user interface integration embedding a Power App or PowerBI report.
Application integration
Application integration is used for integrating an existing third-party application so that it supports consistent working methods and data consistency requirements. Application integration is characterised by navigation backwards and forwards between Dynamics 365 and the other application during one or more working processes. At the same time, complex data can be transferred in both directions several times during one work process. This can be accomplished using Universal Service Desk.
Press button integration
Press button integration is used in a ’loose’ integration – typical for web applications. During press button integration, data is transferred from Dynamics 365 to another system at a point in the work process when the other system is. The information that is transferred can be used to identify the user and navigate to the correct place in the application so that the work process can continue. In press button integration, no data is put back to Dynamics 365 from the other application.
Data Integration
Data integration involves combining data residing in different sources and providing users with a unified view of them. This process becomes significant in a variety of situations where data from another system is used to trigger actions within Dynamics 365.
Process Integration
Business Process Integration (BPI) is the synchronisation of a company’s internal operations with those of its other divisions and its trading partners by connecting disparate systems in real-time.
BPI allows for automation of business processes, integration
of systems and services, and the secure sharing of data across numerous
applications. Overcoming integration challenges will enable organisations to
connect systems internally and externally. Power Automate is a prime example
where Dynamics 365 can be used to trigger actions in a multitude of different
systems.
Integration Pattern Technology
User Interface Integration
1st Party Integrations
Microsoft provides several 1st part integrations within Dynamics 365. The primary one to be considered when it comes to data is PowerBI.
Power BI is a business analytics service by Microsoft. It aims to provide interactive visualisations and business intelligence capabilities with an interface simple enough for end-users to create reports and dashboards utilising both Dynamics 365 and other data sources.
PowerApps
PowerApps can be connected to virtually any data source that you can think of, and the data can then be used within a canvas app. Canvas apps can then be seamlessly embedded within Dynamics 365.
Virtual Entities
Virtual entities enable the integration of data residing in external systems by transparently representing that data as entities in Common Data Service, without replication of data and often without custom coding. The initial implementation of this feature provides just read-only support for such entities and has several other limitations. Besides the restrictions listed below, virtual entities behave the same as other custom entities.
Virtual entities replace previous client-side and server-side approaches to integrating external data, which required customised code and suffered from numerous limitations, including imperfect integration, data duplication, or extensive commitment of development resources. Also, for administrators and system customizers, the use of virtual entities dramatically simplifies administration and configuration.
A virtual entity is a definition of an entity in the Common Data Service platform metadata without the associated physical tables for entity instances created in the Common Data Service database. Instead, during runtime, when an entity instance is required, its state is dynamically retrieved from the associated external system. Each virtual entity type is associated with a virtual entity data provider and (optionally) some configuration information from an associated virtual entity data source.
The following data providers ship with Common Data Service:
- An OData v4 provider is included with the service and is installed by default.
- An Azure Cosmos DB (formerly Microsoft Document DB) provider is available from AppSource.
Virtual Entity Limitations
- Data is read-only. The virtual entity feature doesn’t support pushing changes made in Common Data Service back to the external system.
- Only organisation-owned entities are supported. The security filtering applied to user-owned entities is not supported. Access to the virtual entity data can be turned on or off for individual users based on their security role. Field-level security is not supported.
- It must be possible to
model the external data as a Common Data Service entity. This means:
- All entities in the external data source must have an associated GUID primary key.
- All entity properties must be represented as Common Data Service attributes. You can use simple types representing text, numbers, option sets, dates, images, and lookups.
- You must be able to model any entity relationships in Common Data Service.
- An attribute on a virtual entity cannot be calculated or rollup. Any desired calculations must be done on the external side, possibly within or directed by the data provider.
- Although you can add virtual entity columns as a lookup on a grid or other UI views, you cannot filter or sort based on this virtual entity lookup column.
- Auditing and change tracking is not supported. These may be implemented within the external data store.
- Virtual entities cannot be enabled for queues.
- Offline caching of values is not supported for virtual entities.
- A virtual entity cannot represent an activity and do not support business process flows.
- Once created, a virtual entity cannot be changed to be a standard (non-virtual) entity. The reverse is also true: a standard entity cannot be converted into a virtual entity.
Unified Service Desk
Unified Service Desk for Dynamics 365 model-driven apps provides a configurable framework for quickly building applications that agents can get a unified view of the customer data stored in the Common Data Service platform or other applications. You can aggregate customer information from different areas in the Common Data Service platform and third-party systems into an integrated desktop that provides a 360° view of the customer interactions. This gives agents immediate access to business-critical information so they can quickly engage with customers and address queries and issues.
Unified Service Desk, which is built using the User Interface Integration (UII) framework, is designed as a series of adapters and modules that facilitate the management of UI elements (such as pages and dialogues), automatic loading of related records, agent scripting, a configurable toolbar, and so on. Unified Service Desk can be configured and administered using the Common Data Service platform or Microsoft Dynamics 365 for Outlook. Using the Unified Service Desk to configure agent applications doesn’t require you to write code for the most part, and therefore reduces the lead time to design an agent application as per your business requirements. Also, with the computer telephony integration (CTI) framework of UII, organisations can build adapters to connect Unified Service Desk with their existing CTI infrastructure to support customer communication in agent desktops over various channels such as chat, email, or telephone.
Push Button Integration
This is by far the easiest way to integrate within the UI. At its purest form, a button can launch a third-party application. Other options include passing a parameter to open the correct location of the third-party application.
Data Integrations
Azure Data Factory
The Azure Data Factory (ADF) is a service designed to allow developers to integrate disparate data sources. It is a platform somewhat like SSIS in the cloud to manage the data you have both on-prem and in the cloud.
It provides access to on-premises data in SQL Server and cloud data in Azure Storage (Blob and Tables) and Azure SQL Database. Access to on-premises data is enabled through a data management gateway that connects to on-premises SQL Server databases or other data sources.
Integration paths can be easily created using the visual editor and the available connectors. Customer connectors can also be designed if required. The data can be integrated either in real-time or via batch.
Power Automate
Microsoft Power Automate lets you create automated processes between your favourite apps and services. The ability to run flows from within model-driven apps in Dynamics 365, such as Dynamics 365 Sales and Customer Service, make it simple for users to combine a broad spectrum of services that can be initiated from within Dynamics 365 apps, such as messaging, social engagement, and document routing services.
As Power Automate has over 300 connectors, it can be used to provide data integration between multiple systems. A multitude of functions are available to transform data during integration.
If the specific connector you require is not available, then custom connectors can be created.
Logic Apps
Before considering Logic Apps as an option there are two simple questions to ask yourself.
- Do you have extensive Azure experience?
- Do you live in the Azure portal?
If you answer no to these questions then there is no reason to look at Logic Apps. The functionality between Power Automate and Logic apps are comparable and only getting closer every day.
Kingswaysoft
This high-performance, codeless and easy-to-use data integration solution enables integration between Microsoft Dynamics 365 applications and virtually any other application or data system.
The SSIS Integration Toolkit for Microsoft Dynamics 365 includes top-end features such as support for Bulk API, advanced filtering capabilities, multithreaded writing, multiple write actions (Create, Update, Delete, Upsert, as well as 365 CE/CRM, CDS/PowerApps specific Merge, Convert and ExecuteWorkflow), advanced filtering, and methods to handle incremental changes easily.
Data Export Service
Data Export is an add-on service made available as a Common Data Service solution that adds the ability to replicate Common Data Service data to a Microsoft Azure SQL Database store in a customer-owned Microsoft Azure subscription. The supported target destinations are Microsoft Azure SQL Database and Microsoft Azure SQL Server on Microsoft Azure virtual machines. Data Export intelligently synchronises the entire Common Data Service schema and data initially and after that, synchronises continuously as changes occur (delta changes) in Common Data Service. It must be noted that the data is only one way from Dynamics 365 and the Common Data Service to Azure SQL there is no functionality to write backwards.
Data Integration Considerations – API Limits
To ensure consistent availability and performance for everyone, Microsoft applies some limits to how APIs are used. They limit the number of concurrent connections per user account, the number of API requests per connection, and the amount of execution time that can be used for each connection. These are evaluated within a five-minute sliding window. When one of these limits is exceeded, an exception will be thrown by the platform.
Service protection API limits help ensure that users running applications cannot interfere with each other based on resource constraints. The restrictions will not affect regular users of the platform. Only applications that perform a large number of API requests may be affected. The limits provide a level of protection from random and unexpected surges in request volumes that threaten the availability and performance characteristics of the Common Data Service platform.
Since plug-ins and custom workflow activities execute on the server independent of a logged-on user, API calls made from plug-in code will not count against service protection API limits.
Any data integration can only access Dynamics 365 or Common Data Service through the Web API
More details in regards to the API limits can is available here: https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/api-limits
Process Integration
Power Automate
Power Automate is a great business process automation tool that has over 300 connectors at the time of writing. You can easily continue or trigger a business process automation outwith Dynamics 365 using the functionality and connectors available.
Third-Party Business Process Tools
There are a number of different business process tools available that perform in a similar way to Power Automate. These should only be looked at if there is a specific limitation within Power Automate that stops you from delivering a particular requirement. This however, should not be the case as to where Power Automate is unable to achieve something out of the box you have the option of custom connectors or Azure functions available. Power Automate would always be my goto option as it sits within the
