Azure HDInsight
An Azure managed cluster service for open-source analytics.
210 questions
1 answer
How can I make user defined parameters required inside a pipeline
If I have a parameter that I defined, how can I make it required like this
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 50%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
1 answer
HDInsight cluster creation- error on configuration+pricing
Hi, I have been trying to create an HDInsight Cluster from Canada, but it fails at the configuration step. I subscribed for PAYG, I tried by selecting different nodes but none of them is working it gives me the error " You have reached your…
asked 2022-10-10T23:29:36.653+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 70%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
ANJU RANI
1
Reputation point
commented 2022-10-17T06:48:35.18+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
How do I find Hive server information
I'm trying to create a pipeline to copy data from csv to a DataBricks table. To do so, I believe I need to set up a HIVE linked service. However, I'm not sure where I can find the necessary information to fill out the LS form - we had a…
asked 2022-09-14T14:27:33.613+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 33%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPO%3C/text%3E%3C/svg%3E)
Peter Ott
1
Reputation point
commented 2022-10-10T07:00:21.857+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
0 answers
Azure HDInsight Spark job is failing with Logger Error
Hello Team, Our jobs are recently failing with this error- ERROR RawSocketSender [MdsLoggerSenderThread]: org.fluentd.logger.sender.RawSocketSender java.net.SocketException: Broken pipe (Write failed) All these pyspark jobs were running fine…
asked 2022-09-28T16:10:09.787+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 75%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Aishwarya Gopikrishnan
1
Reputation point
commented 2022-10-04T16:32:49.827+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,081
Reputation points
1 answer
One of the answers was accepted by the question author.
Real Case Scenarios
Hello, Where can I find case scenarios or real life use cases of for example cloud models or high availability and scalability. What I mean is, for example, hybrid cloud is used by banks because they want to control the database and security. …
asked 2022-09-28T09:43:19.933+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 95%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rawan Ghalayini
21
Reputation points
commented 2022-10-03T07:20:57.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 4%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
ShaktiSingh-MSFT
15,056
Reputation points
1 answer
One of the answers was accepted by the question author.
How to use UA Managed Identity in Data factory On Demand HD Insight Linked Service
When creating an on-demand HD Insight linked service, there's missing detail for how to configure a User Assigned managed identity instead of a service principal. Steps are shown on how to add a UA managed identity to the Data Factory, but what values…
asked 2022-09-21T22:36:31.707+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 76%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES3%3C/text%3E%3C/svg%3E)
Scoot-3223
91
Reputation points
commented 2022-09-23T01:37:38.683+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
30,891
Reputation points Microsoft Employee
2 answers
Spark Dataframe writing issue in azure from spark: One of the request inputs is not valid
I am able to read data from azure blob storage but when writing back to azure storage then it throws below error . I am running this program in my local machine. Can someone help me out on this please. Program val config = new SparkConf(); …
asked 2022-08-25T04:24:33.64+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 6%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAH%3C/text%3E%3C/svg%3E)
Abdul Hafiz A.G A ID(RITM0203509)
11
Reputation points
answered 2022-09-14T00:02:23.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 1%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Junjie Cao
1
Reputation point Microsoft Employee
0 answers
How to Add a subqueue in yarn
I already have queues setup on Yarn on HdInsight, they were setup with the Ambari UI. I have a queue for sqoop that takes up 70% of the cluster. However I have a few huge sqoop jobs with a lot of mappers that take up 100% of the queue and block…
asked 2022-08-10T19:52:34.937+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 4%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Vamsi Anamaneni
1
Reputation point
commented 2022-09-08T19:06:11.337+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
HDInsight HBase vs Databricks
Hi, this is probably answered or perhaps a tall question. What would be the difference/benefits between using HDInsight HBase vs Databricks. Azure storage is definitely one. If the aim is to have the convenience of traditional table & sql with…
asked 2022-09-01T16:06:11.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 64%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
AKM
1
Reputation point
commented 2022-09-08T08:41:11.13+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
To use Azure Data Lake Storage Gen2 with Azure HDInsight clusters, do I have to attach the storage to clusters as linked additional storage?
To use Azure Data Lake Storage Gen2 with Azure HDInsight clusters, do I have to attach the storage to clusters as linked additional storage? Or as long as permission is granted to managed identity, spark scala application could access the storage using…
asked 2022-08-30T20:05:46.367+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 32%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Summer
1
Reputation point
commented 2022-09-08T08:33:39.86+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
2 answers
Creating a HDInsight Spark 4.0 cluster with managed identity and a Data Lake Store gen 2 storage account
Hi! I am trying to create a HDI cluster with an ADLS Gen2 storage account as primary storage account. I have created multiple containers inside my storage account, and I want to limit the access of the managed identity to the other containers. …
asked 2022-08-16T12:48:09.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 59%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELC%3C/text%3E%3C/svg%3E)
LHIND, CARSTEN
1
Reputation point
commented 2022-08-29T10:53:43.143+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
HDInsights cluster is in the Error status, even when the user assigned Managed Identity is assigned a role as Storage Blob Data Owner.
HDInsights cluster is in the Error status, even when the user assigned Managed Identity is assigned a role as Storage Blob Data Owner.
asked 2022-08-05T19:40:37.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 99%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Akash Chopra
36
Reputation points
commented 2022-08-22T05:17:00.84+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
HD insight Cluster, Worker node, E32_V3 (256 GB), memory issue
HD insight Cluster - Worker node - E32_V3 (256 GB). It is showing 911 GB memory on ambari portal. MS document for E32_v3 of HD insight worker node showing 1600 GB space. Why there is a discrepancy.
asked 2022-06-15T06:46:45.273+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 94%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETP%3C/text%3E%3C/svg%3E)
Thakur, Prabhat
81
Reputation points
accepted 2022-07-25T14:12:00.207+00:00
Thakur, Prabhat
81
Reputation points
1 answer
One of the answers was accepted by the question author.
Unable to create HDInsight cluster through free azure subscription
There are not enough cores available to support the selected number of nodes. Please adjust the number of nodes selected, pick a different region, or open a support case to request additional HDInsight cores. You have reached your subscription's…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 35%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
1 answer
azure hdinsight There are not enough cores available
I want to create HDInsight in my pay as you go subscription, but I get error: There are not enough cores available to support the selected number of nodes. I checked in my subscription usage and quotas for computing and usage is for every processor…
asked 2022-07-06T21:41:26.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 84%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Ales Ventus
46
Reputation points
commented 2022-07-11T05:39:06.973+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Files not getting saved in Azure blob using Spark in HDInsights cluster
We've setup HDInsights cluster on Azure with Blob as the storage for Hadoop. We tried uploading files to the Hadoop using hadoop CLI and the files were getting uploaded to the Azure Blob. Command used to upload: Hadoop fs -put somefile…
asked 2022-06-14T11:44:29.873+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 94%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Saif Ahmad
21
Reputation points
commented 2022-06-21T13:09:24.653+00:00
Saif Ahmad
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Connect Synapse Spark Pool with Kafka on HDInsight
I have created a Kafka on HDinsight cluster . I have also created Azure Synapse Analytics - Spark Pool on same region as HDinsight. I need guidance on how to consume topics from Kafka into Spark Structured Streaming. Any documentation or steps will be of…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 44%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
HDInsight - Kafka - Version 3.2
Hi all Is there a roadmap to release a cluster with a higher kafka version than 2.4.1 in the near future? Thanks for the info in advance. Best reagrads, Michael
asked 2022-05-30T13:56:47.557+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 49%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Michael Ahrens
21
Reputation points
accepted 2022-06-07T09:26:48.037+00:00
Michael Ahrens
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Can Azure Streaming Analytics read from Kafka on HDInsight and write to Deltalake table on Synapse lake.
Hello I am looking for guidance on building a new event driven platform. The options we are exploring for processing are - Azure Stream Analytics Apache Spark Structured Streaming in Synapse Source is like going to be Kafka on HDInsight …
1 answer
what is the best way to copy data from my hadoop on prem cluster to the azure hdinsight cluster?
hi experts, what is the best way to copy data from my hadoop on prem cluster to the azure hdinsight cluster? So we recently deployed a new hdinsight cluster and now I would like to copy some data from my onprem cluster to hdinsight. Thanks,
asked 2022-05-16T20:57:25.917+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 80%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERY%3C/text%3E%3C/svg%3E)
Richmond Yu
1
Reputation point
commented 2022-06-01T05:47:08.503+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee