Microsoft has released SQL Server 2017 CTP 2.1 for preview, excited to try it out.
- Now available on Linux as well along with Windows.
- SQL Server Integration Services on Linux
- Updated SQL Server Tooling – mssql-scripter tool, DBFS tool
- New lightweight installer for SQL Server Reporting Services
Download it now: https://www.microsoft.com/en-us/evalcenter/evaluate-sql-server-2017-ctp/
Most common challenges faced while managing large data:
- How to maintain historical data.
- How to handle data changes.
- Impact on having triggers on table to capture every single change in data for auditing.
- How to recover accidental changes.
- Calculating trends over time.
To address these challenges Microsoft has introduced a system-versioned temporal table, a new type of user table in SQL Server 2016. A temporal tables are designed to maintain a full history of data changes which allows you to find the state of data at any point in time. This is completely manage by database engine. When you create Temporal table system creates two table 1- Current table & 2. History table.
- FileTable & FILESTREAM features are not supported.
- CASCADE option can be used in case of referencing tables.
- INSTEAD OF triggers are not supported, though AFTER triggers are supported.
Every temporal table has two explicit defined columns with a datetime2 data type (period columns), that you can use as start (SysStartTime) and end (SysEndTime) periods for which row id valid.
How to create Temporal Table:
How to query Temporal Table:
Managing Data Retention:
Data grows very fast when we track each & every change from transnational table, historical data. So how long we should keep the data available in history table and how to move it out of table when retention period expires. Retention period can be decided based on business requirement, however to move data we have multiple options:
An interesting topic to learn more about. Refer to msdn:- https://docs.microsoft.com/en-us/sql/relational-databases/tables/temporal-tables
Microsoft has announced the public preview of disaster recovery for Azure IaaS virtual machines (VMs) using Azure Site Recovery (ASR). With this new capability of Azure Backup for IaaS VM, you can also protect IaaS based applications running on Azure by replicating on to different Azure regions.
- Offered “as-a-Service” – No need of additional infrastructure.
- Simplified experience – Enables Cross-region DR.
- Application-aware security – Control a fail over.
- Non-disruptive DR drills
What is Polybase?
Polybase is a technology that accesses and combines both non-relational and relational data, all from within SQL Server. It allows you to run queries on external data in Hadoop or Azure blob storage.
- It does not require to install additional software.
- You can leverage T-SQL syntax for querying external data.
- No knowledge about HADOOP or Azure is required.
- Access data in Hadoop database with T-SQL.
- Access Azure blob storage with T-SQL.
- Import data from Hadoop or blobs as regular SQL Server tables.
- Export data to Hadoop or Azure blob storage.
- Integrate well BI tools – SSRS, SSAS, PowerPivot, PowerQuery, PowerView, Tableau, Microstrategy or Cognos.
PolyBase uses external tables to access data in Azure blob storage. Since data is not stored within Azure SQL DW, PolyBase handles authentication to external data by using a database-scoped credential.
Using PolyBase in Azure SQL DW Steps:
- Create Master Key – to encrypt secret of your database scoped credential
CREATE MASTER KEY;
- Create Database Scoped Credential – to specify authentication information for your Azure storage account.
CREATE DATABASE SCOPED CREDENTIAL AzureStorageCredential
IDENTITY = ‘polybasepoc’,
SECRET = ‘secretkeygenerated’
- Create External Data Source – to specify the location of your Azure blob storage.
CREATE EXTERNAL DATA SOURCE AzureStorage
TYPE = HADOOP,
LOCATION = ‘wasbs://firstname.lastname@example.org’,
CREDENTIAL = AzureStorageCredential
- Create External File Format – to specify the format of your data.
CREATE EXTERNAL FILE FORMAT DelimitedText
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS (FIELD_TERMINATOR = ‘[||]’, STRING_DELIMITER = ‘[~~]’),
DATA_COMPRESSION = ‘org.apache.hadoop.io.compress.GzipCodec’
- Create External Table – to specify the table definition and location of the data.
CREATE EXTERNAL TABLE ExternalSource_CC (
How to use?
select * from ExternalSource_CC
Limitations of Polybase:
- PolyBase works with defined structure specified in External Table & File Format.
- PolyBase does not allow to exclude Text Qualifiers.
- PolyBase provides row number for failed records but does not give cell level failure information.
- PolyBase does not recognize delimiter within data and in case delimiter is repeated in text, fails to read data.
- Polybase only works with delimited text file, no other tabular format supported except Hadoop.
- At present Polybase supports loading data files that have been UTF-8 encoded.
What is Azure SQL Data Warehouse?
Azure SQL Data Warehouse is PaaS based cloud solution for data storage. It is a massively parallel processing (MPP) cloud-based, scale-out, relational database capable of processing massive volumes of data.
Why you should consider moving to SQL Data Warehouse?
– Massively parallel processing solution. This makes queries many times faster than SQL DB.
– Combine relational data with cloud scale-out capabilities.
– Handles large volume data with massive storage. No storage limit.
– Processing power (compute) can be increase / decrease processing on one click within few seconds.
– Pause or resume DW on-demand, helps in cost saving when not in use.
– Supports SQL Server T-SQL language
– Platform as a service (PaaS) cloud based solution.
– Manages, detect and mitigate security threats Azure authentication features.
– High availability, gives you 99.9% up-time SLA in regions available to public.
– Built-in automated database backups.
– Minimal maintenance, reduced dependency on IT team.
– Good for OLAP environment.
Why you should not consider moving to SQL Data Warehouse?
– Supports SQL Server T-SQL language (not everything though).
– Supports only 32 concurrent connections.
– Supports only 1024 active connections.
– Does not support in-memory OLTP.
– Does not support CROSS database queries.
– Migration from on-premise or IaaS to SQL DW is a challenge.
– Blocking issues may take down entire Azure SQL DW.
– INSERT BULK API is not supported thoroughly.
– Not good for OLTP environment due to frequent changes.
Reference Link :- https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-overview-what-is
Azure Cosmos DB (“Project Florence”) revolutionary way for globally replicating your data, multi-model database service. Azure Cosmos DB lets you store your data globally-distributed and elastically scale throughput and storage across any number of geographical regions at anytime with single click. Addition to this, Azure Cosmos DB automatically indexes your data which helps in fast querying & also guarantees millisecond latencies. On 10th May 2017 Microsoft announces general availability of Azure Cosmos DB.
Checkout msdn blog for detailed info : https://azure.microsoft.com/en-us/blog/a-technical-overview-of-azure-cosmos-db/
Is there any widely used SQL coding standard out there? Here it goes…
Link : – SQL Coding Guidelines