Back to overview
Resolved

Oasis gRPC (grpc.oasis.io) and Emerald Web3 Gateway (emerald.oasis.io) are down

Nov 7, 2024 at 12:23am UTC

Resolved
Nov 7, 2024 at 10:02pm UTC

Requests to grpc.oasis.io, sapphire.oasis.io, and emerald.oasis.io were intermittently timing out due to load balancer misconfigurations and Oasis node-specific issues.

To resolve the issues we have reconfigured load balancer pools to include more other production nodes and removed problematic nodes. We have also restarted affected components (e.g., proxies) and updated the Linux kernel to address filesystem inconsistencies on impacted nodes and restore stability.

Services are now operational, and traffic is routing correctly. In the future, we plan to stabilize nodes by removing some unneeded components and enhancing monitoring to detect these kinds of issues.

Updated
Nov 7, 2024 at 9:21pm UTC

Oasis gRPC (grpc.oasis.io) and Emerald Web3 Gateway (emerald.oasis.io) recovered.

Updated
Nov 7, 2024 at 8:59pm UTC

Oasis gRPC (grpc.oasis.io) and Emerald Web3 Gateway (emerald.oasis.io) went down.

Updated
Nov 7, 2024 at 12:01pm UTC

Oasis gRPC (grpc.oasis.io) recovered.

Updated
Nov 7, 2024 at 12:00pm UTC

Oasis gRPC (grpc.oasis.io) went down.

Updated
Nov 7, 2024 at 6:13am UTC

Oasis gRPC (grpc.oasis.io) recovered.

Updated
Nov 7, 2024 at 6:08am UTC

Oasis gRPC (grpc.oasis.io) went down.

Updated
Nov 7, 2024 at 5:59am UTC

Oasis gRPC (grpc.oasis.io) recovered.

Updated
Nov 7, 2024 at 5:47am UTC

Oasis gRPC (grpc.oasis.io) went down.

Updated
Nov 7, 2024 at 5:13am UTC

Emerald Web3 Gateway (emerald.oasis.io) recovered.

Updated
Nov 7, 2024 at 4:40am UTC

Emerald Web3 Gateway (emerald.oasis.io) went down.

Updated
Nov 7, 2024 at 4:22am UTC

Oasis gRPC (grpc.oasis.io) recovered.

Updated
Nov 7, 2024 at 3:45am UTC

Oasis gRPC (grpc.oasis.io) went down.

Updated
Nov 7, 2024 at 3:24am UTC

Emerald Web3 Gateway (emerald.oasis.io) recovered.

Updated
Nov 7, 2024 at 3:18am UTC

Oasis gRPC (grpc.oasis.io) recovered.

Updated
Nov 7, 2024 at 3:14am UTC

Oasis gRPC (grpc.oasis.io) went down.

Updated
Nov 7, 2024 at 3:08am UTC

Oasis gRPC (grpc.oasis.io) recovered.

Updated
Nov 7, 2024 at 2:43am UTC

Emerald Web3 Gateway (emerald.oasis.io) went down.

Updated
Nov 7, 2024 at 2:32am UTC

Emerald Web3 Gateway (emerald.oasis.io) recovered.

Updated
Nov 7, 2024 at 2:21am UTC

Oasis gRPC (grpc.oasis.io) went down.

Updated
Nov 7, 2024 at 1:56am UTC

Oasis gRPC (grpc.oasis.io) recovered.

Updated
Nov 7, 2024 at 1:56am UTC

Emerald Web3 Gateway (emerald.oasis.io) went down.

Updated
Nov 7, 2024 at 1:40am UTC

Emerald Web3 Gateway (emerald.oasis.io) recovered.

Updated
Nov 7, 2024 at 1:40am UTC

Emerald Web3 Gateway (emerald.oasis.io) went down.

Updated
Nov 7, 2024 at 1:35am UTC

Emerald Web3 Gateway (emerald.oasis.io) recovered.

Updated
Nov 7, 2024 at 1:01am UTC

Oasis gRPC (grpc.oasis.io) went down.

Updated
Nov 7, 2024 at 12:37am UTC

Oasis gRPC (grpc.oasis.io) recovered.

Updated
Nov 7, 2024 at 12:26am UTC

Oasis gRPC (grpc.oasis.io) went down.

Created
Nov 7, 2024 at 12:23am UTC

Emerald Web3 Gateway (emerald.oasis.io) went down.