Infinity Bots - Lịch sử thông báo

Trải qua hiệu suất bị giảm sút một phần

96% - thời gian hoạt động

Production - Đang hoạt động

96% - thời gian hoạt động
thg 5 2025 · 93.9135%thg 6 · 93.4766%thg 7 · 100.000%
thg 5 2025
thg 6 2025
thg 7 2025

Development - Đang hoạt động

97% - thời gian hoạt động
thg 5 2025 · 96.7115%thg 6 · 94.9102%thg 7 · 100.000%
thg 5 2025
thg 6 2025
thg 7 2025

Documentation - Đang hoạt động

99% - thời gian hoạt động
thg 5 2025 · 100.000%thg 6 · 95.5282%thg 7 · 100.000%
thg 5 2025
thg 6 2025
thg 7 2025

Staff Panel - Đang hoạt động

97% - thời gian hoạt động
thg 5 2025 · 95.4099%thg 6 · 95.3602%thg 7 · 100.000%
thg 5 2025
thg 6 2025
thg 7 2025

Widgets - Đang hoạt động

93% - thời gian hoạt động
thg 5 2025 · 83.6022%thg 6 · 95.0940%thg 7 · 100.000%
thg 5 2025
thg 6 2025
thg 7 2025
94% - thời gian hoạt động

Production - Hiệu suất giảm sút

91% - thời gian hoạt động
thg 5 2025 · 89.1873%thg 6 · 85.2706%thg 7 · 100.000%
thg 5 2025
thg 6 2025
thg 7 2025

Development - Đang hoạt động

96% - thời gian hoạt động
thg 5 2025 · 94.1470%thg 6 · 94.8889%thg 7 · 100.000%
thg 5 2025
thg 6 2025
thg 7 2025
100% - thời gian hoạt động

Cloudflare → Always Online - Đang hoạt động

Cloudflare → DNS Firewall - Đang hoạt động

Cloudflare → DNS Root Servers - Đang hoạt động

Cloudflare → DNS Updates - Đang hoạt động

Cloudflare → Firewall - Đang hoạt động

Cloudflare → Gateway - Đang hoạt động

Cloudflare → Network - Đang hoạt động

Cloudflare → Pages - Đang hoạt động

Discord → API - Đang hoạt động

Discord → Gateway - Đang hoạt động

Github → Actions - Đang hoạt động

Github → API Requests - Đang hoạt động

Github → Git Operations - Đang hoạt động

Github → Issues - Đang hoạt động

Github → Pull Requests - Đang hoạt động

Github → Webhooks - Đang hoạt động

Ionos → Cloud Backup - Đang hoạt động

Ionos → Cloud Server - Đang hoạt động

Stripe → Stripe API - Đang hoạt động

Vercel → Xây dựng - Đang hoạt động

Vercel → Build & Deploy - Đang hoạt động

Vercel → DNS - Đang hoạt động

Lịch sử thông báo

thg 7 2025

Không có thông báo nào được báo cáo trong tháng này

thg 6 2025

Server Downtime
  • Sau khi chết
    Sau khi chết

    📝 Postmortem: June 2025 Service Outage

    Incident Duration:

    June 16, 2025 – June 21, 2025

    Status: Resolved

    Root Cause: Misconfigured infrastructure and networking components

    📌 Summary

    Between June 16 and June 21, 2025, our services experienced a prolonged and critical disruption. This impacted system accessibility, network stability, and overall deployment reliability. The root causes were traced back to multiple misconfigurations within our new infrastructure stack, primarily involving our Dokploy instance, networking setup, and reverse proxy (Traefik).

    ⚙️ Technical Cause

    Upon investigation, we identified several compounding issues:

    • Misconfigured Dokploy Instance: The initial deployment lacked critical network isolation and routing configurations, leading to service timeouts and container miscommunication.

    • Traefik Reverse Proxy: Misconfigured routing and TLS handling caused failed ingress connections and prevented external traffic from reaching internal services.

    • Networking Setup Errors: Overlapping subnets and improperly bridged networks led to intermittent connectivity between deployer and host machines, further destabilizing the system.

    • Missing Health Checks: Some containers were not being monitored properly, which delayed automatic restarts and extended service downtime.

    🚑 Immediate Actions Taken

    • Isolated deployer and host networks to stabilize inter-service traffic.

    • Corrected routing rules and middleware configuration in Traefik.

    • Rebuilt the Dokploy configuration with clearer network separation and improved error handling.

    • Re-enabled and audited health checks across services.

    • Conducted live testing and verification to ensure full service restoration by June 21, 2025.

    ✅ Resolution and Recovery

    The system was gradually stabilized beginning on June 16, with partial access restored within 30 minutes of our first major fix. However, additional network-level issues prolonged the resolution timeline. By June 21 at 3:35 AM, all services were fully restored and verified functional.

    📚 Lessons Learned

    • Configuration reviews must be enforced before production deployment of new infrastructure tools.

    • Network planning (IP ranges, bridges, proxies) needs to be documented and peer-reviewed.

    • Critical systems (like DNS routing, ingress, and orchestration layers) must have dedicated monitoring and rollback plans.

    🔧 Preventative Measures

    • Implement automated preflight checks in our deployment pipelines.

    • Schedule recurring audits of proxy and ingress configurations.

    • Build fallback container orchestration playbooks for Dokploy-based deployments.

    • Expand post-deploy smoke testing to catch network-level regressions earlier.

    🗣️ Final Note

    We sincerely apologize for the extended downtime and the impact it had on your experience. While our intention was to modernize our infrastructure, we recognize that our transition planning and oversight fell short. This will be addressed internally, and improvements are already underway.

    Thank you for your patience and continued support.

  • Đã khắc phục
    Đã khắc phục
    This incident has been resolved.
  • Cập nhật
    Cập nhật

    We apologize that these issues have been ongoing for so long, our team is still working to resolve the issues! And properly configure our new dokploy network!

  • Cập nhật
    Cập nhật

    Some additional issues have arised and we are working on a fix

  • Cập nhật
    Cập nhật

    We have isolated networks between our deployer and host machine to help stable out its long term usage, our team is currently finishing up with the final setup stages. Services should start being restored within the next 20 - 30 mins.

  • Đã nhận diện
    Đã nhận diện

    We have identified this issue is due to Dokploy. We are working resolving this issue now.

thg 5 2025

thg 5 2025 đến thg 7 2025

Sau