[V2] edgeHub just stopped working? #558
Comments
|
Hello @Hammatt Thanks for the information. Do you know which version of the edgeHub you're running? It will be at the top of the starting banner, Like this: I'm also curious about the iotedgectl setup command line (minus the connection string), if you're willing to share. |
|
Hi @darobs , I'm not able to see any logs in the edge hub at all, what I posted in the original post is the entire output of This is what we're using for the setup command: Edit: possibly relevant to mention, all other modules start up (but time out because they can't connect to the hub) |
|
Just some more information, I've been trying to verify ways to reproduce. So far: This doesn't seem to happen on windows 10 Pro, the module starts up and everything works fine. But on every Windows 10 IoT Core based device that I've tested this on, the issue occurs. Hope this is of use. |
|
not sure if this is relevant, but I am also seeing edgeHub keep on start/stop. docker logs -f edgeHub results: I couldn't see version information thru docker logs -f edgeHub, below is result of docker logs -f edgeAgent result of docker images |
|
This seems to be ARM-specific, we will be investigating. |
|
Hey @darobs, I can confirm that this is not limited only to ARM as the Windows 10 IoT Core Devices that we have are all x64 architecture. Specifically we've been able to reproduce on a Minnowboard turbot dual Ethernet Quad-Core (Intel Atom E3845) model, and a number of other x64 based devices. I don't have access to any ARM devices to test this on. |
|
Got my problems of the day mixed up... The problem @michael-chi is seeing on the Raspberry Pi has been fixed and should be pushed out to Docker. @Hammatt - we're still looking at this problem. |
|
Thanks @darobs , is there a way that I could roll back to a working version here? It's blocking me pretty hard at work. |
|
Here's what I would try in the following order:
If this is the Windows networking issue we're seeing, the first should fix the problem. |
|
That first command isn't working: Edit: Neither is the 2nd: |
|
My bad. That command works as is in Linux, and I thought the same form worked in Powershell. Essentially, you want to run ...and the other mistake is it's not |
|
Alright, so what I've done is stop all the containers, run I then pruned again and tried with the --image flag this time. I wasn't quite sure what to put after the image flag as you said a couple of different things but i went with I then check and it has the same output again: Not really sure what's going on now, if it happens in this version too. Have you been able to reproduce the issue at all? |
|
You have to rollback to preview21.
If you don’t put the image info you will get preview22.
…Sent from my phone
On Mar 28, 2018, at 4:55 PM, Alexander Hammatt ***@***.***> wrote:
Alright, so what I've done is stop all the containers, run docker system prune -a and then double checked that the containers are gone with docker ps -a. Then I've ran the setup command without the image argument and the result was the same:
docker logs edgeHub
docker :
+ CategoryInfo : NotSpecified: (:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Unhandled Exception: System.AggregateException: One or more errors occurred. (Access is denied) --->
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Access is denied
at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle
password, X509KeyStorageFlags keyStorageFlags)
at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password,
X509KeyStorageFlags keyStorageFlags)
at Microsoft.Azure.Devices.Edge.Hub.Service.Hosting.Initialize(String certPath) in
/opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Hosting.cs:line 23
at Microsoft.Azure.Devices.Edge.Hub.Service.Program.<MainAsync>d__1.MoveNext() in
/opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in
/opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 27
I then pruned again and tried with the --image flag this time. I wasn't quite sure what to put after the image flag as you said a couple of different things but i went with microsoft/azureiotedge-agent:1.0.0-preview021, and not the one for the hub because it seemed to change the Edge Agent Image field that the setup command displayed.
I then check and it has the same output again:
docker logs edgeHub
docker :
+ CategoryInfo : NotSpecified: (:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Unhandled Exception: System.AggregateException: One or more errors occurred. (Access is denied) --->
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Access is denied
at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle
password, X509KeyStorageFlags keyStorageFlags)
at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password,
X509KeyStorageFlags keyStorageFlags)
at Microsoft.Azure.Devices.Edge.Hub.Service.Hosting.Initialize(String certPath) in
/opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Hosting.cs:line 23
at Microsoft.Azure.Devices.Edge.Hub.Service.Program.<MainAsync>d__1.MoveNext() in
/opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in
/opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 27
Not really sure what's going on now, if it happens in this version too. Have you been able to reproduce the issue at all?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
edit: oh, i think i see what you mean now |
|
Sorry for the confusion earlier, I'm up and running now on edgeHub Version - 1.0.0-preview021.10543704 |
|
Hi, I was curious, is there anyone that can successfully run preview22 version of edgeAgent , edgeHub using windows container? I have run into the same issue as @Hammatt, and I've tried both Windows 10 destop, Windows IoT Core, but all failed with the same exception. Can we say latest edgeHub windows container (privew22) has blocking issue, and the only way to start IoT Edge on windows platform is to rollback to preview21? And what's the steps to rollback to preview21? By running iotedgectl setup to change edgeAgent image version doesn't make change to edgeHub version, it still use the latest edgeHub image (preview22)... Any help would be very appreciated, thanks! |
|
@yphuangms , you have to change it in the Azure portal. Go to where you would set your modules. Click on Configure advanced Edge runtime settings. And change image to: |
|
I was able to run preview22 error free with Windows containers on my Windows 10 PC, but it was a completely new deployment. |
|
@Orlando1991, Thanks! It helps a lot! And from trial and error, I found that only edgeHub requires rollback. @darobs , Do you know if there will be a quick release to recover edgeHub preview22? Or just leave it as it is, those who encounter the same issue have to resolve on their own? |
|
I can confirm that new version of edge runtime works fine with RPi now. |
|
Hey @darobs, I can run preview22 with windows containers on windows 10 pro, the problem occurs for me on Windows 10 IoT Core only. When you say that you were able to run it, was that windows 10 pro or windows 10 iot core? |
|
Windows 10 pro, not IoT Core. I've reached out to our Windows experts for more help. |
|
After running |
|
@Hammatt do you by chance use SetMethodHandlerAsync and/or SetMethodDefaultHandlerAsync? |
|
@v-tbert no |
|
Hello @Hammatt According to our Windows experts, this looks like a permissions issue on the certificate file. They suggested possibly a missing read or read/execute permission. Would you please check this? |
|
Hey @darobs , I should have permission as I'm running the commands from an admin account. ([Security.Principal.WindowsPrincipal] `
[Security.Principal.WindowsIdentity]::GetCurrent()
).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)returns true. If you're able to tell me the location on disk where the certificate could be, I could double check. |
|
Just updating, still seeing this issue here. Edge Hub Preview022 won't start up on any of our iot core devices, but downgrading to Preview 021 without changing anything else does start up. |
We've been developing an application built on IoT Edge, I just restarted the runtime and the edgeHub module will no longer start up.
Edit:
I've tried restarting the device. it's running windows 10 iot core if that's of any relevance. I've re-ran the setup command a few times to see if that changed anything and it hasn't.
The text was updated successfully, but these errors were encountered: